Service scaling method and device

Abstract

The present disclosure relates to the technical field of computers, and provides a service scaling method and device. The method comprises: allocating an original allocation partition and a to-be-allocated partition of each container based on a preset idempotent service allocation partition rule and a container number of the service, wherein the idempotent service allocation partition rule describes a relationship between a partition number and a number of containers started by the service; determining a partition to be migrated out, a partition to be migrated in, and corresponding migration data of each; performing data migration between different containers based on the partition to be migrated out, the partition to be migrated in, and the corresponding migration data of each container; determining a non-renaming partition and a to-be-renamed partition and corresponding directory modification data for each container after data migration, and modifying a directory name of the to-be-renamed partition according to the directory modification data. The technical solution of the embodiment of the present disclosure can reduce data migration and reduce the impact of service scaling on a business system.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, and particularly relates to a service scaling method and device. BACKGROUND

[0002] At present, big data applications are booming, and in the occasions of camera snapshot collection, lower cascade, video stream analysis and other occasions requiring computing of massive feature data, service distributed storage based on cloud architecture is widely used.

[0003] In the service distributed storage system based on the cloud architecture, data is distributedly stored in each service, and the data volume is huge. When the service needs to be scaled to adapt to the change of the access business data volume, how to ensure that the history data of up to 10 billion huge data volume is not lost and available, how to ensure that the distributed service has correct data shards, and how to reduce data migration and reduce the impact on the business system are urgent problems to be solved in the service scaling of distributed storage data.

[0004] In the related art, when the distributed storage data service is scaled, a consistent hashing algorithm can be designed, and the data is bound to the service when stored, and the data migration is determined according to the algorithm when the service is scaled. This scheme can effectively realize the correct migration of data when the service is scaled, but the number of migrations is completely related to the algorithm, which will result in a huge amount of data migration, thereby affecting the business of the business system. SUMMARY

[0005] Therefore, the embodiments of the present disclosure provide a service scaling method and device, an electronic device and a computer readable storage medium to solve the problem of huge data migration when the service is scaled in the prior art.

[0006] In a first aspect, the embodiments of the present disclosure provide a service scaling method, which comprises: allocating an original allocation partition and a to-be-allocated partition of each container based on a preset idempotent service allocation partition rule and a container number of the service to obtain an allocation result, wherein the idempotent service allocation partition rule describes the relationship between the partition number and the number of containers started by the service; determining, for each container, a partition to be migrated out, a partition to be migrated in and corresponding migration data based on the allocation result; performing data migration between different containers based on the partition to be migrated out, the partition to be migrated in and the corresponding migration data of each container; determining, for each container after data migration, a non-renaming partition that does not need to modify a directory name, a to-be-renamed partition that needs to modify the directory name and corresponding directory modification data, and modifying the directory name of the to-be-renamed partition according to the directory modification data.

[0007] In a second aspect, the embodiment of the present disclosure provides a service scaling device, which comprises: a partition allocation module, configured to allocate original allocation partitions and to-be-allocated partitions of each container based on a preset idempotent service allocation partition rule and a container number of the service, to obtain an allocation result, wherein the idempotent service allocation partition rule describes a relationship between a partition number and a number of containers started by the service; a data determination module, configured to determine, for each container, to-be-migrated partitions, to-be-migrated partitions and corresponding migration data based on the allocation result; a data migration module, configured to perform data migration between different containers based on the to-be-migrated partitions, the to-be-migrated partitions and the corresponding migration data of each container; and a directory change module, configured to determine, for each container after data migration, non-renaming partitions that do not need to modify a directory name and to-be-renamed partitions that need to modify the directory name and corresponding directory modification data, and modify the directory name of the to-be-renamed partitions according to the directory modification data.

[0008] In a third aspect, the embodiment of the present disclosure provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above method when executing the computer program.

[0009] In a fourth aspect, the embodiment of the present disclosure provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0010] Compared with the prior art, the embodiment of the present disclosure has the beneficial effects that the original allocation partitions and to-be-allocated partitions of the containers of the service are calculated according to the number of containers started by the service, and the to-be-migrated partitions and to-be-renamed partitions of the containers are determined according to the changes of the original allocation partitions and to-be-allocated partitions, so as to perform data migration and partition renaming, which can reduce data migration as much as possible, reduce inter-disk and network data transmission, and thus maximally reduce the impact of service scaling on the business system. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 is a flowchart of a service scaling method provided by the embodiment of the present disclosure;

[0013] Figure 2 is a flowchart of another service scaling method provided by the embodiment of the present disclosure;

[0014] Figure 3 is a structural schematic diagram of a service scale-out device provided by an embodiment of the present disclosure.

[0015] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the present embodiments of the present disclosure. However, persons skilled in the art should understand that the present disclosure can be implemented in other embodiments without these specific details. In other cases, well-known systems, devices, circuits, and methods have not been described in detail in order not to obscure the description of the present disclosure with unnecessary details.

[0017] In the related art, the approximate implementation process of the stateful distributed service storing data based on the StatefulSet deployment is as follows:

[0018] The topic of kafka is created as N partitions, and each service establishes a directory to store data according to the partition, and the data is uniformly distributed depending on the load balancing capability of kafka, wherein N is a natural number, which can be set as needed, for example, N can be set as 60. The directory name of the directory established by the service is the service number, which can be P+partition number, for example, P0, P1, etc.

[0019] The stateful service based on the StatefulSet deployment designs an idempotent algorithm to allocate partitions to services with different numbers according to the service number, the number of kafka topic partitions, and the number of services to be started. Among them, the kafka container (POD) data of the service needs to have a corresponding serial number, that is, the service name, and the number starts from 0. For example, the person service needs to start 5 containers (PODs), and the service name can be: person-0, person-1, person-2, person-3, and person-4.

[0020] Further, the service specifies the partition to consume kafka data and store the full data consumed by itself.

[0021] Based on the above implementation process of the stateful distributed service storing data, since the data is distributedly stored in each service, and the data volume is huge, when the service needs to be scaled out to adapt to the change of the access data volume, the general problem of scaling out the distributed storage data service needs to be solved urgently.

[0022] To solve the above problems in the prior art, an embodiment of the present disclosure provides a service scale-out solution.

[0023] A service scaling method and device according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 is a flowchart of a service scaling method provided by an embodiment of the present disclosure. The method provided by the embodiment of the present disclosure can be executed by any electronic device with computer processing capability, such as a terminal or a server. As shown in the figure, the service scaling method comprises the following steps. Figure 1

[0025] In step S101, original allocation partitions and to-be-allocated partitions of each container are allocated based on a preset idempotent service allocation partition rule and a container number of the service, to obtain an allocation result, wherein the idempotent service allocation partition rule describes a relationship between a partition number and a container number of the service.

[0026] Specifically, idempotency is a common problem in a distributed environment, and the results of multiple operations of an idempotent operation are consistent in the distributed environment. The idempotent service allocation partition rule of the embodiment of the present disclosure is a service allocation partition rule that meets the idempotency requirement, which can calculate a partition number of a partition allocated to a container according to a container number of the service and other variables. For a single container, the original allocation partition refers to a partition before service scaling, and the to-be-allocated partition refers to a partition after service scaling.

[0027] In the embodiment of the present disclosure, the scaling change of the service can be determined according to the change of the number of containers started by the service. If the number of containers started by the service increases, the service is scaled up; if the number of containers started by the service decreases, the service is scaled down.

[0028] In step S102, the to-be-migrated partitions, the to-be-migrated-in partitions and the corresponding migration data of each container are determined based on the allocation result.

[0029] Specifically, according to the comparison between the original allocation partition and the to-be-allocated partition of a single container, it can be determined which partitions in the original allocation partition of the current container need to be migrated, to obtain migration data of the to-be-migrated partitions.

[0030] In step S103, data migration between different containers is performed based on the to-be-migrated partitions, the to-be-migrated-in partitions and the corresponding migration data of each container.

[0031] In step S104, the non-renaming partitions that do not need to modify the directory name, the to-be-renamed partitions that need to modify the directory name and the corresponding directory modification data of each container after data migration are determined, and the directory name of the to-be-renamed partition is modified according to the directory modification data.

[0032] ​Specifically, after data migration between different containers is performed, part of the partitions that do not need to be migrated and part of the partitions that need to be migrated need to be renamed.

[0033] According to the partition data balancing technical solution provided in the embodiments of the present disclosure, the existing kafka partition load balancing capability is used, data is stored in a partition directory in a service, and a kind of idempotent algorithm is designed to balance the partitions corresponding to the service. When the service is scaled, data migration is not needed for all partitions, but through dynamic calculation of the binding relationship between the container and the partition, the to-be-migrated partition is determined through comparison between the originally allocated partition and the to-be-allocated partition, and the partition directory name to be modified is determined, so that the amount of data migration can be reduced, and the impact on the business system is minimized.

[0034] In the embodiments of the present disclosure, the idempotent service partition allocation rule can include the following formula (1):

[0035] (1)

[0036] wherein, P represents a partition number, n represents a current container number, SN represents a number of containers started by the service, KPN represents a number of kafka topic partitions, and x represents an integer greater than or equal to 0 and less than (KPN-n) / SN. For the service name person-0, the container number is 0. SN corresponds to the value of spec.replicas of Statefulset, and KPN is a value determined when the kafka topic is created.

[0037] The formula (1) takes x as a variable to describe the partition number of the current container with the container number n, and the partition number has a correlation with the number of containers started by the service.

[0038] Based on the above formula (1), the rule for the to-be-allocated partition list of the container with the container number n is the following formula (2):

[0039] (2)

[0040] As shown in Figure 2 , a service scaling method according to an embodiment of the present disclosure includes the following steps:

[0041] Step S201, using the idempotent service partition allocation rule to calculate the partitions used by the current containers.

[0042] Specifically, the formula (1) is used to calculate the partitions owned by the current containers and the information about the partitions to be owned by the target containers.

[0043] Step S202, calculating the number of partitions to be migrated by each container.

[0044] Specifically, the number of partitions that each container needs to migrate can be obtained by subtracting the target number of container partitions from the current number of container partitions. A positive value indicates that the partitions need to be migrated out, and a negative value indicates that the partitions need to be migrated in.

[0045] In step S203, it is determined whether the service is scaled up or scaled down.

[0046] According to the change in the number of containers started by the service, it can be determined whether the service is scaled up or scaled down. If the service is scaled up, step S204 is performed, and if the service is scaled down, step S205 is performed.

[0047] In step S204, the partitions that do not need to be migrated are calculated.

[0048] In step S205, the partitions that need to be migrated are calculated.

[0049] Specifically, the partitions that need to be migrated can be calculated according to the partitions that do not need to be migrated obtained in step S204. In step S205, a list of containers that need to migrate out data is obtained and sorted in ascending order according to the container number; a list of containers that need to migrate in data is obtained and sorted in ascending order according to the container number; the partition information owned by the current container list that needs to migrate out is taken out, and the partitions that do not need to be migrated and do not need to be renamed after scaling up or down are filtered out, the partition information obtained in step S205 is sorted, and the number of partitions that need to be migrated determined in step S202 is taken from the largest partition number.

[0050] In step S206, the partitions that need to be migrated in from one or more containers are calculated.

[0051] Specifically, the target service of the migrated data is calculated by obtaining the list of services that need to be migrated in.

[0052] In step S207, the partition data migration is performed.

[0053] Specifically, when it is confirmed that the current container is a container that needs to migrate out data, the entire partition is migrated using rsync. Rsync is a data mirroring backup tool.

[0054] In step S208, the directory name information that needs to be modified is calculated.

[0055] Specifically, the partitions that do not need to modify the directory name are filtered, and the correspondence between the original directory name and the target directory name of the partitions that need to modify the directory name is calculated.

[0056] In step S209, the partition directory is modified.

[0057] Specifically, the directory name modification operation of the partition that needs to modify the directory name is performed.

[0058] After step S209 is executed, the directory name modification operation of all containers is completed, and the service is restarted to complete the service scaling.

[0059] The service scaling solution of the embodiments of the present disclosure is described in detail below for the service scaling and service scaling cases respectively.

[0060] In the embodiments of the present disclosure, if the number of containers started by the service increases, steps S102 and S103 perform the service scaling operation.

[0061] Correspondingly, in step S102, the first number of partitions that each container needs to migrate data after the number of containers started by the service increases can be obtained, and the partitions that do not need to migrate data and the partitions that do not need to be renamed of each container are determined according to the original allocation partition and the should allocation partition of each container, and the partitions that need to migrate data and the corresponding migration data of each container are determined according to the first number of partitions, the partitions that do not need to migrate data and the partitions that do not need to be renamed, wherein the partitions that need to migrate data include the partitions to be migrated out and the partitions to be migrated in.

[0062] Further, in step S102, when determining the partitions that need to migrate data and the corresponding migration data of each container, the partition numbers of the partitions other than the partitions that do not need to migrate data and the partitions that do not need to be renamed of each container can be sorted, the first number of partitions with the largest partition number sequence are selected as the partitions to be migrated out, and the partition numbers of the partitions to be migrated out are sorted, and the to-be-migrated-in partition of the container to be migrated in is determined according to the sequence of the partition number and the sequence of the container number. Wherein the first number is determined according to the first number of partitions.

[0063] Wherein, when determining the to-be-migrated-in partition of the container to be migrated in according to the sequence of the partition number and the sequence of the container number, the target migration-in container of the partition to be migrated out can be determined as the container to be migrated in with the smaller sequence of the container number in sequence from small to large according to the sequence of the partition number and the sequence of the container number. The target migration-in container of the partition to be migrated out is the migration target of the data of the partition to be migrated out.

[0064] Further, in step S102, when determining the to-be-renamed partition and the corresponding directory modification data, the directory number of the partition that does not need to be renamed can be obtained, the first partition number of the partition of the container to be migrated in is determined according to the partitions that need to migrate data and the corresponding migration data, and the to-be-renamed partition and the corresponding directory modification data are determined according to the directory number of the partition that does not need to be renamed, the first partition number, and the sorting of the directory numbers of the original allocation partition and the should allocation partition of the container.

[0065] In the embodiments of the present disclosure, if the number of containers started by the service increases, steps S102 and S103 perform the service scaling operation.

[0066] Correspondingly, in step S102, the second number of partitions that each container needs to migrate after the number of containers started by the service is reduced can be obtained, and the partitions that each container needs to migrate data and the partitions that do not need to be renamed are determined according to the second number of partitions, the original allocation partitions and the to-be-allocated partitions of each container, and the partitions that each container needs to migrate data and the corresponding migration data are determined according to the partitions that need to migrate data and the partitions that do not need to be renamed, and the to-be-renamed partitions and the corresponding directory modification data are determined.

[0067] Further, in step S102, when determining the partitions that each container needs to migrate data and the corresponding migration data, the container numbers and the partition numbers of each container can be sorted, the second number of partitions with larger container numbers and larger partition numbers are selected as the partitions to be migrated out, and the to-be-migrated-in partitions of the containers to be migrated in are determined according to the sequence numbers of the partition numbers and the sequence numbers of the container numbers. The second number is determined according to the second number of partitions.

[0068] In the step S102, when determining the to-be-migrated-in partitions of the containers to be migrated in according to the sequence numbers of the partition numbers and the sequence numbers of the container numbers, the target migrated-in containers of the partitions to be migrated out can be determined in sequence from small to large according to the sequence numbers of the partition numbers and the sequence numbers of the container numbers, as the containers to be migrated in with smaller sequence numbers of the container numbers.

[0069] Further, in step S102, when determining the to-be-renamed partitions and the corresponding directory modification data, the second partition numbers of the partitions of the containers to be migrated in can be determined according to the partitions that need to migrate data and the corresponding migration data, and the to-be-renamed partitions and the corresponding directory modification data are determined according to the sorting of the directory numbers of the partitions that do not need to be renamed, the second partition numbers, and the directory numbers of the original allocation partitions and the to-be-allocated partitions of the containers.

[0070] In the following embodiments, the partition data balancing technical solution in the service scaling change in the present disclosure is demonstrated by taking collecting human body data as an example.

[0071] In the following embodiments, the number of KPN of the service theme of collecting human body is set to 60.

[0072] Correspondingly, the corresponding partition numbers of the service theme of collecting human body are: [0, 1, 2, 3, 4, 5, 6, 7…57, 58, 59], and the corresponding directories are: [P0, P1, P2, P3, P4, P5, P6, P7…P57, P58, P59].

[0073] At present, the number of containers started by the human body service is 3, and the container numbers are respectively: [0 (person-0), 1 (person-1), 2 (person-2)].

[0074] At this time, each container allocation partition is as follows:

[0075] The partition number corresponding to the storage partition number of container number 0 is: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57].

[0076] The partition number corresponding to the storage partition number of container number 1 is: [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58].

[0077] The partition number corresponding to the storage partition number of container number 2 is: [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59].

[0078] After expanding the 3 containers of the human service to 5 containers, the new container number is: [0 (person-0), 1 (person-1), 2 (person-2), 3 (person-3), 4 (person-4)]. After the service expansion, the final requirement for each container allocation partition is as follows:

[0079] The partition number corresponding to the storage partition number of container number 0 is: [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55].

[0080] The partition number corresponding to the storage partition number of container number 1 is: [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56].

[0081] The partition number corresponding to the storage partition number of container number 2 is: [2, 7, 12, 17, 22, 27, 32, 37, 42, 47, 52, 57].

[0082] The partition number corresponding to the storage partition number of container number 3 is: [3, 8, 13, 18, 23, 28, 33, 38, 43, 48, 53, 58].

[0083] The partition number corresponding to the storage partition number of container number 4 is: [4, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59].

[0084] After shrinking the 3 containers of the human service to 2 containers, the new container number is: [0 (person-0), 1 (person-1)]. After the service shrinkage, the final requirement for each container allocation partition is as follows:

[0085] The partition numbers corresponding to the container numbers 0 of the storage partitions are: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58].

[0086] The partition numbers corresponding to the container numbers 1 of the storage partitions are: [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59].

[0087] The following is an example of calculating the partitions that a container should be assigned, with the number of kafka partitions (KPN) being 60, the number of containers (SN) of the human service being 5, and the current container number (n) being 3:

[0088] 0 <= x < (KPN-n) / SN

[0089] == > 0 <= (60-3) / 5

[0090] == > 0 <= x < 11.4

[0091] == > x can take values [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

[0092] Substitute x with values [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] into formula (1) in turn, and the results are as follows:

[0093] P(3,0) = 0*5 + 3 = 3

[0094] P(3,1) = 1*5 + 3 = 8

[0095] P(3,2) = 2*5 + 3 = 13

[0096] P(3,3) = 3*5 + 3 = 18

[0097] P(3,4) = 4*5 + 3 = 23

[0098] P(3,5) = 5*5 + 3 = 28

[0099] P(3,6) = 6*5 + 3 = 33

[0100] P(3,7) = 7*5 + 3 = 38

[0101] P(3,8) = 8*5 + 3 = 43

[0102] P(3,9) = 9*5 + 3 = 48

[0103] P(3, 10) = 10*5 + 3 = 53

[0104] P(3, 11) = 11*5 + 3 = 58

[0105] Based on formula (2), the partition number of the partition to which the container numbered 3 should be allocated is:

[0106] P(3) = [P(3, 0), P(3, 1), P(3, 2), P(3, 3), P(3, 4), P(3, 5), P(3, 6), P(3, 7), P(3, 8), P(3, 9), P(3, 10), P(3, 11)] = [3, 8, 13, 18, 23, 28, 33, 38, 43, 48, 53, 58].

[0107] The following is the migration data algorithm derivation of expanding the 3 containers of human services to 5 containers:

[0108] The final data distribution of the current 3 containers of human services is derived using formulas (1) and (2) as follows:

[0109] The partition numbers corresponding to the storage partitions of container number 0 are: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57].

[0110] The partition numbers corresponding to the storage partitions of container number 1 are: [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58].

[0111] The partition numbers corresponding to the storage partitions of container number 2 are: [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59].

[0112] The final data distribution of the expanded 5 containers of human services is derived using formulas (1) and (2) as follows:

[0113] The partition numbers corresponding to the storage partitions of container number 0 are: [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55].

[0114] The partition numbers corresponding to the storage partitions of container number 1 are: [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56].

[0115] The partition number of the storage partition corresponding to container number 2 is: [2, 7, 12, 17, 22, 27, 32, 37, 42, 47, 52, 57].

[0116] The partition number of the storage partition corresponding to container number 3 is: [3, 8, 13, 18, 23, 28, 33, 38, 43, 48, 53, 58].

[0117] The partition number of the storage partition corresponding to container number 4 is: [4, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59].

[0118] The derivation process for the partitions that need to be migrated for each container numbered to expand to 5 human service containers is as follows:

[0119] First, calculate the number of partitions that each service needs to migrate:

[0120] When the human service is in 3 containers, each container is allocated 20 partitions. After expansion, each container is allocated 12 partitions. The number of partitions in the original container (Sn) with the same container number is subtracted from the number of partitions in the destination container (Dn). The original number can be the container number when there are fewer containers. When the container with the corresponding number does not exist, the number of partitions is 0.

[0121] S0 = S0 - D0 = 20 – 12 = 8

[0122] S1 = S1 - D1 = 20 – 12 = 8

[0123] S² = S² - D² = 20 – 12 = 8

[0124] S3=S3 (does not exist)-D3=0–12=-12

[0125] S4 = S4 (does not exist) - D4 = 0 - 12 = -12

[0126] Where S0, S1... represent the partition that should be migrated from container number n, Sn represents the partition that should be allocated from the original container number n, and Dn represents the partition that should be allocated from the destination container number n. Positive numbers indicate that the partition should be migrated out, and negative numbers indicate that the partition should be migrated in.

[0127] Next, the partitions that do not require migration or renaming are calculated. The partition numbers of each container before expansion are:

[0128] S0: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57]

[0129] S1 : [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58]

[0130] S2 : [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59]

[0131] S3 : []

[0132] S4 : []

[0133] Wherein, the containers numbered S3 and S4 do not exist, and the number of partitions thereof is 0, thus the set is empty.

[0134] According to the original allocated partitions and the partitions to be allocated, the partition numbers of the partitions which do not need to be migrated out or renamed can be found, that is:

[0135] S0 : [0, 15, 30, 45]

[0136] S1 : [1, 16, 31, 46]

[0137] S2 : [2, 17, 32, 47]

[0138] At this time, the number of partitions currently possessed by the containers numbered 0, 1 and 2 is greater than the number of partitions which should be possessed by the containers, thus the larger partition numbers can be migrated out.

[0139] In the calculation of the partitions to be migrated out, the original container filters out the partition numbers of the partitions which do not need to be migrated out or renamed, uses ascending order sorting of the partitions, and intercepts the largest partition number to migrate out, and the partitions to be migrated out are [33, 36, 39, 42, 48, 51, 54, 57] of S0, [34, 37, 40, 43, 49, 52, 55, 58] of S1 and [35, 38, 41, 44, 50, 53, 56, 59] of S2.

[0140] Subsequently, the partitions to be migrated in from a certain Sn are calculated, and the calculation method is to migrate the smallest number of the smallest number of the containers to the smallest number of the partitions to be migrated in after sorting the partitions.

[0141] Specifically, when the data needs to be migrated in and out of the containers, the containers are sorted in ascending order, and the containers with smaller numbers are preferentially allocated to the containers with smaller numbers of the partitions to be migrated in, until all the allocation is completed.

[0142] At this time, the final allocation result is: S0: [33, 36, 39, 42, 48, 51, 54, 57] and S1: [34, 37, 40, 43] are allocated to S3, and S1: [49, 52, 55, 58] and S3: [35, 38, 41, 44, 50, 53, 56, 59] are allocated to S4.

[0143] According to the above allocation result, data migration can be performed. When performing data migration, migrating the entire directories P0, P1, etc. can migrate P33, P36, P39, P42, P48, P51, P54, P57 in S0 to S3, P34, P37, P40, P43 in S1 to S3, P49, P52, P55, P58 in S1 to S4, and P35, P38, P41, P44, P47, P50, P53, P56, P59 in S2 to S4, thereby obtaining the partition numbers of the expanded containers as follows:

[0144] S0: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 45]

[0145] S1: [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 46]

[0146] S2: [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 47]

[0147] S3: [33, 36, 39, 42, 48, 51, 54, 57, 34, 37, 40, 43]

[0148] S4: [49, 52, 55, 58, 35, 38, 41, 44, 50, 53, 56, 59]

[0149] After that, for the partition numbers of the expanded containers, the partition directories that do not need to be renamed are calculated, and it can be obtained that P33, P48 in S3 do not need to be modified, and P44, P59 in S4 do not need to be modified. In addition, the partition numbers of the partitions that do not need to be migrated out or renamed in the above are also considered, i.e. S0: [0, 15, 30, 45], S1: [1, 16, 31, 46], and S2: [2, 17, 32, 47].

[0150] After obtaining all the partition directories that do not need to be renamed, when recording the modification of the partition directory names, the partitions that do not need to be renamed are filtered, sorted by partition size, and the smallest partition number is calculated to obtain the modification relationship from the original partition directory name to the destination partition directory name, which is modified as follows:

[0151] S0: [3->5, 6->10, 9->20, 12->25, 18->35, 21->40, 24->50, 27->55]

[0152] S1: [4->6, 7->11, 10->21, 13->26, 19->36, 22->41, 25->51, 28->56]

[0153] S2: [5->7, 8->12, 11->22, 14->27, 20->37, 23->42, 26->52, 29->57]

[0154] S3: [36->8, 39->18, 42->28, 51->43, 54->53, 57->58, 34->3, 37->13, 40->23, 43->38]

[0155] S4: [49->19, 52->29, 55->39, 58->54, 35->4, 38->9, 41->14, 50->24, 53->34, 56->49]

[0156] The following is the migration data algorithm derivation of reducing 3 containers of human services to 2 containers:

[0157] The final data distribution of the current 3 containers of human services is derived using formulas (1) and (2) as follows:

[0158] The partition numbers corresponding to the storage partition of container number 0 are: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57].

[0159] The partition numbers corresponding to the storage partition of container number 1 are: [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58].

[0160] The partition numbers corresponding to the storage partition of container number 2 are: [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59].

[0161] The final data distribution of reducing to 2 containers of human services is derived using formulas (1) and (2) as follows:

[0162] The partition numbers corresponding to the container numbers 0 of the storage partition are: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58].

[0163] The partition numbers corresponding to the container numbers 1 of the storage partition are: [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49, 51, 53, 55, 57, 59].

[0164] The derivation process of the partitions that each numbered container needs to migrate for the container reduction to 2 human service containers is as follows:

[0165] First, the number of partitions that each service needs to migrate is calculated:

[0166] When the human service is 3 containers, each container is allocated 20 partitions, and after reduction, each container is allocated 30 partitions. The number of original container partitions (Sn) of the same container number is subtracted from the number of destination container partitions (Dn), wherein the container number in the state of fewer containers can be taken as the original number, and when the corresponding numbered container does not exist, the corresponding number of partitions is 0.

[0167] S0=S0-D0=20–30=-10

[0168] S1=S1-D1=20–30=-10

[0169] S2=S2-D2 (non-existent)=20–0=20

[0170] Wherein, S0, S1… indicates the partitions that the container number n should migrate, Sn indicates the partitions that the original container number n should allocate, Dn indicates the partitions that the destination container number n should allocate, positive number indicates migration out, and negative number indicates migration in.

[0171] In the calculation of the partitions that need to be migrated out, according to the final data distribution of the current 3 human service containers, the partitions are sorted in ascending order of partition number, and the maximum partition number is migrated out. The partitions that need to be migrated out are S2 [2, 5, 8, 11, 14, 17, 20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59] whole partitions.

[0172] Then, the partitions that need to be migrated in from a certain Sn are calculated. The calculation method is to migrate the minimum number of partitions of the minimum number of containers to the minimum number of partitions that need to be migrated in after sorting the partitions.

[0173] In particular, when data needs to be migrated into and out of the containers, the containers are sorted in ascending order of the container numbers, and the containers and partitions with smaller numbers are preferentially allocated to the containers with smaller numbers that need to be migrated into the partitions, until all the allocations are completed. At this time, the final allocation result is that [2, 5, 8, 11, 14, 17, 20, 23, 26, 29] of S2 is migrated to S0, and [32, 35, 38, 41, 44, 47, 50, 53, 56, 59] of S2 is migrated to S1.

[0174] According to the above allocation result, data migration can be performed. When the data migration is performed, the entire directories P0, P1, etc. are migrated, so that P2, P5, P8, P11, P14, P17, P20, P23, P26, P29 of S2 are migrated to S0, and P32, P35, P38, P41, P44, P47, P50, P53, P56, P59 of S2 are migrated to S1, thereby obtaining the partition numbers of the containers after the shrinkage as follows:

[0175] S0: [0, 3, 6, 9, 12, 15, 18, 21, 24, 27, 30, 33, 36, 39, 42, 45, 48, 51, 54, 57, 2, 5, 8, 11, 14, 17, 20, 23, 26, 29]

[0176] S1: [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46, 49, 52, 55, 58, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59]

[0177] Subsequently, for the partition numbers of the containers after the shrinkage, the directories of the partitions that do not need to be renamed are calculated first, and it can be obtained that [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 2, 8, 14, 20, 26] in S0 and [1, 7, 13, 19, 25, 31, 37, 43, 49, 55, 35, 41, 47, 53, 59] in S1 do not need to be renamed.

[0178] After obtaining all the directories of the partitions that do not need to be renamed, when the partition directory names are recorded and modified, the partitions that do not need to be renamed are filtered, sorted according to the partition size, and the smallest partition number is calculated to obtain the modification relationship from the original partition directory name to the target partition directory name, and the modification is as follows:

[0179] S0: [3->4, 9->16, 15->28, 21->34, 27->40, 33->46, 39->50, 45->52, 51->56, 57->58, 5->10, 11->22, 17->32, 23->38, 29->44]

[0180] S1: [4->3, 10->5, 16->9, 22->11, 28->15, 34->21, 40->27, 46->33, 52->45, 58->57, 32->17, 38->23, 44->29, 50->39, 56->51]

[0181] By adopting the technical solutions of the embodiments of the present disclosure, when the stateful distributed storage service is scaled, the historical data can be ensured not to be lost and available. The technical solutions ensure that the partitions obtained by service restart are consistent by designing an idempotent algorithm, and achieve the technical effects of reducing the amount of migrated data and reducing the impact on the business system.

[0182] According to the service scaling method, the original allocation partition and the to-be-allocated partition of the containers of the service are calculated according to the number of containers started by the service, and the to-be-migrated partition and the to-be-renamed partition of the containers are determined according to the changes of the original allocation partition and the to-be-allocated partition, so as to perform data migration and partition renaming, which can reduce data migration as much as possible, reduce inter-disk and network data transmission, and thus maximally reduce the impact of service scaling on the business system.

[0183] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure. The service scaling apparatus described below can be referred to in correspondence with the service scaling method described above. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0184] Figure 3 is a schematic diagram of a service scaling apparatus provided by an embodiment of the present disclosure. As shown in Figure 3 The service scaling apparatus includes:

[0185] The partition determination module 301 is a partition allocation module, configured to allocate the original allocation partition and the to-be-allocated partition of each container based on a preset idempotent service partition allocation rule and the container number of the service, to obtain an allocation result, wherein the idempotent service partition allocation rule describes the relationship between the partition number and the number of containers started by the service.

[0186] The data determination module 302 is configured to determine the to-be-migrated partition, the to-be-migrated-in partition, and the corresponding migration data of each container based on the allocation result.

[0187] The data migration module 303 is configured to perform data migration between different containers based on the to-be-migrated partition, the to-be-migrated-in partition, and the corresponding migration data of each container.

[0188] The directory change module 304 is configured to determine, for each container after data migration, non-renaming partitions that do not need to modify directory names and renaming partitions that need to modify directory names and corresponding directory modification data, and modify the directory names of the renaming partitions according to the directory modification data.

[0189] In the embodiments of the present disclosure, if the number of containers started by the service increases, the data determination module 302, the data migration module 303, and the directory change module 304 perform service scaling operations.

[0190] Specifically, the data determination module 302, when performing the service scaling operation, is specifically configured to: obtain a first partition quantity of data that needs to be migrated for each container after the number of containers started by the service increases; determine, according to the allocation result, partitions of each container that do not need to migrate data and non-renaming partitions; and determine, according to the first partition quantity, the partitions of each container that do not need to migrate data and the non-renaming partitions, partitions of each container that need to migrate data and corresponding migration data, wherein the partitions that need to migrate data include partitions that need to be migrated out and partitions that need to be migrated in.

[0191] Further, when determining the partitions of each container that need to migrate data and corresponding migration data and the renaming partitions and corresponding directory modification data, the data determination module 302 can sort the partition numbers of the partitions other than the partitions of each container that do not need to migrate data and the non-renaming partitions, select a first quantity of partitions with the largest partition number sequence as the partitions that need to be migrated out, wherein the first quantity is determined according to the first partition quantity, and sort the partition numbers of the partitions that need to be migrated out, and determine, according to the sequence of the partition numbers and the sequence of the container numbers, the to-be-migrated-in partitions of the containers that need to migrate data.

[0192] Further, when determining, according to the sequence of the partition numbers and the sequence of the container numbers, the to-be-migrated-in partitions of the containers that need to migrate data, the data determination module 302 can determine, according to the sequence of the partition numbers and the sequence of the container numbers, the target migration-in containers of the partitions that need to be migrated out as the containers that need to migrate data with the sequence of the container numbers from small to large in turn.

[0193] When determining, for each container after data migration, non-renaming partitions that do not need to modify directory names and renaming partitions that need to modify directory names and corresponding directory modification data, the directory change module 304 can obtain the directory numbers of the non-renaming partitions; determine first partition numbers of the partitions of the containers that need to migrate data according to the partitions that need to migrate data and the corresponding migration data; and determine, according to the directory numbers of the non-renaming partitions, the first partition numbers, and the sorting of the directory numbers of the original allocation partitions and the to-be-allocated partitions of the containers, the renaming partitions and the corresponding directory modification data.

[0194] In this embodiment of the disclosure, if the number of containers started by the service decreases, the data determination module 302, the data migration module 303, and the directory change module 304 perform a service scaling-down operation.

[0195] Specifically, when performing the service scaling-down operation, the data determination module 302 is used to: obtain the number of second partitions that each container needs to migrate after the number of containers started by the service is reduced; determine the partitions that need to migrate data and the unnamed partitions for each container based on the number of second partitions and the allocation results; and determine the migration data corresponding to the partitions that need to migrate data for each container based on the partitions that need to migrate data and the unnamed partitions.

[0196] In this embodiment of the disclosure, the idempotent service allocation partitioning rule includes the following formula (1):

[0197] (1)

[0198] Where P represents the partition number, n represents the current container number, SN represents the number of containers started by the service, KPN represents the number of Kafka topic partitions, and x represents an integer greater than or equal to 0 and less than (KPN-n) / SN.

[0199] Since the functional modules of the service scaling device in the example embodiments of this disclosure correspond to the steps of the service scaling method in the example embodiments described above, for details not disclosed in the device embodiments of this disclosure, please refer to the embodiments of the service scaling method described above.

[0200] According to the service scaling device of this disclosure, the original allocated partition and the partition to be allocated of the service container are calculated based on the number of containers started by the service, and the partition to be migrated and the partition to be renamed of the container are determined based on the changes of the original allocated partition and the partition to be allocated, so as to perform data migration and partition renaming. This can minimize data migration and reduce data transmission between disks and over the network, thereby minimizing the impact of service scaling on the business system.

[0201] Figure 4 This is a schematic diagram of the electronic device 4 provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module in the various device embodiments described above.

[0202] The electronic device 4 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The electronic device 4 can include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art can understand that Figure 4 The electronic device 4 is merely an example and is not limited to the electronic device 4, and can include more or fewer components or different components than those shown.

[0203] The processor 401 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like.

[0204] The memory 402 can be an internal storage unit of the electronic device 4, for example, a hard disk or a memory of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. The memory 402 can also include both the internal storage unit and the external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0205] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual applications, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0206] The integrated module, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. The computer program can include computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier wave signal, telecommunication signal and software distribution medium, etc.

[0207] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

Claims

1. A service scaling method, characterized in that, The method comprises: obtaining original allocation partitions and to-be-allocated partitions of each container based on preset idempotency service allocation partition rules and container numbers of services, to obtain an allocation result, wherein the idempotency service allocation partition rules describe a relationship between a partition number and a number of containers started by the services, and the idempotency service allocation partition rules comprise the following formula: wherein P represents the partition number, n represents the current container number, SN represents the number of containers started by the services, KPN represents the number of kafka topic partitions, and x represents an integer greater than or equal to 0 and less than (KPN-n) / SN; determining, for each container, to-be-migrated partitions, to-be-migrated-in partitions and corresponding migration data based on the allocation result; performing data migration between different containers based on the to-be-migrated partitions, the to-be-migrated-in partitions and the corresponding migration data of each container; determining, for each container, non-renaming partitions that do not need to modify a directory name and to-be-renamed partitions that need to modify a directory name and corresponding directory modification data after data migration, and modifying the directory name of the to-be-renamed partitions according to the directory modification data.

2. The method of claim 1, wherein, If the number of containers started by the services increases, the step of determining, for each container, to-be-migrated partitions, to-be-migrated-in partitions and corresponding migration data based on the allocation result comprises: obtaining a first number of to-be-migrated data partitions of each container after the number of containers started by the services increases; determining, for each container, partitions that do not need to migrate data and non-renaming partitions based on the allocation result; determining, for each container, to-be-migrated data partitions and corresponding migration data based on the first number of to-be-migrated data partitions, the partitions that do not need to migrate data and the non-renaming partitions, wherein the to-be-migrated data partitions comprise to-be-migrated-out partitions and to-be-migrated-in partitions.

3. The method of claim 2, wherein, The step of determining, for each container, to-be-migrated data partitions and corresponding migration data based on the first number of to-be-migrated data partitions, the partitions that do not need to migrate data and the non-renaming partitions comprises: sorting partition numbers of partitions other than the partitions that do not need to migrate data and the non-renaming partitions of each container, and selecting a first number of partitions with the largest partition number as to-be-migrated-out partitions, wherein the first number is determined according to the first number of to-be-migrated data partitions; sorting the partition numbers of the to-be-migrated-out partitions, and determining to-be-migrated-in partitions of a container that needs to migrate data based on the sequence numbers of the partition numbers and the container numbers.

4. The method of claim 3, wherein, The step of determining to-be-migrated-in partitions of a container that needs to migrate data based on the sequence numbers of the partition numbers and the container numbers comprises: determining, according to the sequence numbers of the partition numbers and the container numbers, target migration-in containers of the to-be-migrated-out partitions as containers that need to migrate data in sequence from small to large in the sequence numbers of the container numbers.

5. The method of claim 2, wherein, The step of determining, for each container, non-renaming partitions that do not need to modify a directory name and to-be-renamed partitions that need to modify a directory name and corresponding directory modification data after data migration comprises: obtaining directory numbers of the non-renaming partitions; determining, according to the partition number of the partition to which data is migrated and the corresponding migration data, a first partition number of a partition of a container to which data is to be migrated; determining, according to the directory number of the non-renamed partition, the first partition number, and the sorting of the original allocated partition and the directory number of the allocated partition of the container, the to-be-renamed partition and the corresponding directory modification data.

6. The method of claim 1, wherein, If the number of containers started by the service decreases, the determining, for each container respectively, of the partition to be migrated out, the partition to be migrated in, and the corresponding migration data based on the allocation result, comprises: obtaining a second partition number to be migrated of each container after the number of containers started by the service decreases; determining, according to the second partition number and the allocation result, the partition to be migrated of each container and the non-renamed partition; determining, according to the partition to be migrated of each container and the non-renamed partition, the migration data corresponding to the partition to be migrated of each container.

7. A service scaling device, characterized by comprising: The apparatus comprises: a partition allocation module configured to allocate, based on a preset idempotent service partition allocation rule and a container number of a service, an original allocated partition and an allocated partition of each container to obtain an allocation result, wherein the idempotent service partition allocation rule describes a relationship between a partition number and a number of containers started by the service, and the idempotent service partition allocation rule comprises the following formula: wherein P represents a partition number, n represents a current container number, SN represents a number of containers started by the service, KPN represents a number of kafka topic partitions, and x represents an integer greater than or equal to 0 and less than (KPN-n) / SN; a data determination module configured to determine, for each container respectively, a partition to be migrated out, a partition to be migrated in, and corresponding migration data based on the allocation result; a data migration module configured to perform data migration between different containers based on the partition to be migrated out, the partition to be migrated in, and the corresponding migration data of each container; a directory change module configured to determine, for each container after data migration, a non-renamed partition for which a directory name does not need to be modified and a to-be-renamed partition for which a directory name needs to be modified, and corresponding directory modification data, and to modify the directory name of the to-be-renamed partition according to the directory modification data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images