Method and apparatus for data migration, electronic device, and storage medium

Abstract

An embodiment of the present application provides a method and apparatus for data migration, an electronic device, and a storage medium, which relate to the field of fintech. The method includes: obtaining the configured number of first instances, and selecting the first instances with the number of first instances from the pre-deployed migration microservice instances with the number of second instances; the number of second instances is greater than the number of first instances; according to the preset proportion adjustment rule, reducing the business proportion of the deployment containers corresponding to the first instances, and increasing the business proportion of the deployment containers corresponding to the second instances that are not selected among the migration microservice instances with the number of second instances; splitting the data to be migrated into data packets with the number of first instances according to the preset business logic; and sending the data packets with the number of first instances to the target database one by one through the corresponding first instances. The embodiment of the present application can enable data migration to meet both the business customization requirements and improve the migration efficiency.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of fintech, and in particular, to a method and apparatus for data migration, an electronic device, and a storage medium. Background Art

[0002] In the field of fintech, with the development of computer technology, more and more financial services use databases to store transaction data. For some financial service operations that have been in use for a long time (such as bank deposits, withdrawals, and transfers), when they were put into use, they did not consider the sharp increase in future transaction data volume. When the database table corresponding to the financial service operation grows to a certain extent, the operation performance of the table drops significantly. At this time, the table has to be migrated. At the same time, to meet the business customization requirements for the data table to be migrated during the migration process, microservices are often used for data migration. In related technologies, for the data migration solution based on microservices, usually a microservice is deployed for data migration. When the amount of data to be migrated is large, the migration efficiency of the microservice is affected by other microservices in the corresponding deployment container, resulting in low migration efficiency. Therefore, there is an urgent need for a data migration method that can meet both business customization requirements and improve migration efficiency. Summary of the Invention

[0003] The main purpose of the embodiments of this application is to propose a method and apparatus for data migration, an electronic device, and a storage medium, so that data migration can meet both business customization requirements and improve migration efficiency.

[0004] To achieve the above object, a first aspect of the embodiments of this application proposes a method for data migration, and the method includes:

[0005] Obtain the configured number of first instances, and select the first instances of the number of first instances from the pre-deployed migration microservice instances of the number of second instances; wherein, the number of second instances is greater than the number of first instances;

[0006] According to a preset proportion adjustment rule, reduce the business proportion of the deployment container corresponding to the first instance, and increase the business proportion of the deployment container corresponding to the second instance that is not selected among the migration microservice instances of the number of second instances;

[0007] According to a preset business logic, split the data to be migrated into data packets of the number of first instances, and the data packets of the number of first instances correspond one-to-one to the first instances of the number of first instances;

[0008] Send the data packets of the number of first instances to the target database one by one through the corresponding first instances.

[0009] In some embodiments, splitting the data to be migrated into a first number of instance data packets according to a preset service logic includes:

[0010] Determining a master table and slave tables in the data to be migrated according to the association relationships of the tables in the data to be migrated;

[0011] Splitting the master table into a first number of instance first data sets;

[0012] Performing data splitting on the slave tables according to the respective first data sets to obtain a plurality of second data sets corresponding one-to-one to the first data sets;

[0013] Integrating each of the first data sets and the corresponding second data sets according to the service logic to obtain a first number of instance data packets.

[0014] In some embodiments, splitting the data to be migrated into a first number of instance data packets according to a preset service logic includes:

[0015] Obtaining a first data size of the data to be migrated;

[0016] Determining a second data size corresponding to each of the data packets according to the first data size and the first number of instances; wherein, the difference between any two of the second data sizes satisfies a preset threshold;

[0017] Splitting the data to be migrated according to the second data size corresponding to each of the data packets to obtain a plurality of the data packets.

[0018] In some embodiments, selecting a first number of first instances from a second number of pre-deployed migration microservice instances includes:

[0019] Sorting the second number of pre-deployed migration microservice instances according to the network parameters of the deployed deployment containers;

[0020] Selecting the first number of first instances at the front from the sorted second number of pre-deployed migration microservice instances.

[0021] In some embodiments, selecting a first number of first instances from a second number of pre-deployed migration microservice instances further includes:

[0022] Allocating identification codes to the respective first instances according to the arrangement order of the respective first instances;

[0023] Correspondingly, sending the first number of instance data packets to a target database one by one through the corresponding first instances includes:

[0024] Associate each of the identification codes with each of the data packets one by one;

[0025] Send each of the data packets to the target database one by one through the first instance corresponding to the associated identification code.

[0026] In some embodiments, the method further includes:

[0027] When there is a faulty first instance, reorder the non-faulty first instances;

[0028] Reassign the identification codes to each of the reordered first instances;

[0029] Determine the distribution code corresponding to the identification code that has not been reassigned according to a preset conversion rule, where the distribution code corresponds to the assigned identification code one by one;

[0030] Send the data packets corresponding to the distribution code to the target database one by one through the corresponding first instance.

[0031] In some embodiments, the method further includes:

[0032] During the migration of the data to be migrated, obtain the updated data to be migrated according to a preset time period;

[0033] Perform data migration on each of the updated data respectively.

[0034] To achieve the above object, a second aspect of the embodiments of the present application proposes an apparatus for data migration, the apparatus includes:

[0035] A selection module, configured to obtain the configured number of first instances, and select the first instances of the number of first instances from the migration microservice instances of the pre-deployed number of second instances; where the number of second instances is greater than the number of first instances;

[0036] An adjustment module, configured to reduce the business proportion of the deployment container corresponding to the first instance according to a preset ratio adjustment rule, and increase the business proportion of the deployment container corresponding to the second instance that has not been selected among the migration microservice instances of the number of second instances;

[0037] A splitting module, configured to split the data to be migrated into data packets of the number of first instances according to a preset business logic, and the data packets of the number of first instances correspond to the first instances of the number of first instances one by one;

[0038] A migration module, configured to send the data packets of the number of first instances to the target database one by one through the corresponding first instance.

[0039] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the method for data migration described in the first aspect above is implemented.

[0040] To achieve the above object, a fourth aspect of the embodiments of the present application provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0041] The method and device for data migration, electronic device, and storage medium provided by the present application select the first number of first instances from the second number of migration microservice instances deployed in advance, so that the second instances do not need to execute data migration. By adjusting the business proportion of the first instances and the second instances, most of the business is processed through the deployment containers corresponding to the second instances, so that stable services can be provided for normal business while reducing the impact of normal business on data migration. At the same time, by setting multiple first instances, the time required for migration can be further compressed. And the first instances integrate business logic, so that customized migration of the migrated data can be achieved. Therefore, compared with the related art, the embodiments of the present application can not only achieve customized migration of the migrated data but also improve the migration efficiency. Description of the Drawings

[0042] Figure 1 is a flowchart of the method for data migration provided by the embodiments of the present application;

[0043] Figure 2 is a schematic diagram of the system framework for data migration in the method for data migration provided by the embodiments of the present application;

[0044] Figure 3 is a schematic diagram of the packet splitting corresponding to an embodiment in the method for data migration provided by the embodiments of the present application;

[0045] Figure 4 is a schematic diagram of the identification code processing when the first instance fails in the method for data migration provided by the embodiments of the present application;

[0046] Figure 5 is a module schematic diagram of the device for data migration provided by the embodiments of the present application;

[0047] Figure 6 is an application schematic diagram of a specific embodiment of the device for data migration provided by the embodiments of the present application;

[0048] Figure 7 is a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present application. Detailed Implementation Manner

[0049] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] It should be noted that although the functional modules are divided in the device schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described may be executed in a different manner from the module division in the device or the sequence in the flowchart. Terms such as "first" and "second" in the description, claims and the above-mentioned drawings are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] First, several nouns involved in the present application are analyzed:

[0053] Microservice: A software development technology - a variant of the service-oriented architecture (SOA) architectural style. It advocates dividing a single application into a group of small services that coordinate and cooperate with each other to provide ultimate value to users. Each service runs in its own independent process, and lightweight communication mechanisms are used to communicate between services (usually RESTful APIs based on HTTP). Each service is built around specific business and can be independently deployed to production environments, pre-production environments, etc.

[0054] In the field of fintech, with the development of computer technology, more and more financial operations use databases to store transaction data. For some financial service operations that have been in use for a long time (such as bank deposits, withdrawals, and transfers), the sharp increase in future transaction data volume was not considered when they were put into use. When the database table corresponding to a financial service operation grows to a certain extent, the operating performance of the table drops significantly. At this time, the table has to be migrated. At the same time, to meet the business customization requirements for the data tables to be migrated during the migration process, such as merging some tables during the migration process, and deleting some data items or data content according to business conditions during the merging, or splitting and reorganizing the data tables again. At this time, microservices are often used to integrate the data to be migrated according to business logic and then perform data migration. However, in related technologies, in the solution for data migration based on microservices, usually one microservice is deployed for data migration. When the amount of data to be migrated is large, the migration efficiency of the microservice is affected by other microservices in the corresponding deployment container, resulting in low migration efficiency. Therefore, there is an urgent need for a data migration method that can meet business customization requirements and improve migration efficiency. Based on this, the embodiments of this application provide a data migration method, device, electronic device, and storage medium, enabling data migration to meet both business customization requirements and improve migration efficiency.

[0055] The data migration method, device, electronic device, and storage medium provided by the embodiments of this application are specifically described through the following embodiments. First, the data migration method in the embodiments of this application is described.

[0056] This application can be used in many general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, small computers, large computers, distributed computing environments including any of the above systems or devices, and so on. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0057] Figure 1 is an optional flowchart of the data migration method provided by the embodiments of this application. Figure 1 The method in may include but is not limited to steps S101 to S104.

[0058] Step S101: Obtain the configured number of first instances, and select the first instances with the number of first instances from the pre-deployed migration microservice instances with the number of second instances; where the number of second instances is greater than the number of first instances.

[0059] It should be noted that the number of first instances can be imported through a configuration file or set through an interface, so as to achieve flexible setting of the number of first instances, and then adapt to more scenarios; for example, during the peak period of normal business processing, dynamically reduce the value of the number of first instances; during the low period of normal business processing, dynamically increase the number of first instances.

[0060] It should be noted that the selection method of the first instances can be based on the hardware parameters of the corresponding deployed containers, or based on migration efficiency or network parameters of the deployed containers, etc. The hardware parameters such as CPU, cache and other hardware configurations, and the network parameters such as IP address, MAC address and transmission bandwidth, etc.; the migration efficiency such as the historical migration data efficiency. In this regard, the embodiments of the present application do not impose too many restrictions here.

[0061] It should be noted that the migration microservice instances refer to the microservices with data migration functions deployed on the deployed containers. The first instances refer to the migration microservice instances with data migration functions and actually performing data migration operations. The second examples refer to the migration microservice instances with data migration functions but not actually performing data migration operations. By setting the migration microservice instances with the number of second instances on multiple deployed containers, more application scenarios can be adapted. For example, after a data migration is completed and the first instances are determined again, when the previous first instances are faulty, at the beginning of each data migration, as many first instances as possible can be selected, and compared with the related technologies, the impact on normal business processing can be reduced.

[0062] Step S102: According to the preset proportion adjustment rule, reduce the business proportion of the deployed containers corresponding to the first instances, and increase the business proportion of the deployed containers corresponding to the second instances that are not selected among the migration microservice instances with the number of second instances.

[0063] It should be noted that the proportion adjustment rule is used to limit the adjustment range of the business proportion between the first instances and the second examples, so that both data migration and normal business run relatively normally. The proportion adjustment rule can be estimated based on the historical business processing capabilities of the corresponding deployed containers, or directly set based on empirical values, or dynamically adjusted based on the business processing situation after the business proportion corresponding to the previous migration. In this regard, the embodiments of the present application do not make restrictions, and those skilled in the art can set the proportion adjustment rule according to the actual situation.

[0064] Step S103: According to the preset service logic, the data to be migrated is segmented into data packets with a first instance quantity, and the data packets with the first instance quantity correspond one by one to the first instances with the first instance quantity.

[0065] It should be noted that the service logic is used to record the conversion relationship of the same type of data between the source database and the target database. For example, the conversion relationship clarifies which data in the data to be migrated is valid data, or the conversion relationship clarifies the change in the data format of the data records before and after migration. In some other embodiments, it may also be the way of combining multiple data tables before migration.

[0066] It should be noted that each data packet includes at least one data record, and the data record is obtained through the conversion of the service logic.

[0067] Step S104: Send the data packets with the first instance quantity to the target database one by one through the corresponding first instances.

[0068] It should be noted that each first instance migrates the corresponding data packet according to the bandwidth of the link where it is located.

[0069] It should be noted that the target database is the final destination for the data to be migrated. After receiving the data packet, the target database will store it to complete the data migration.

[0070] Therefore, by selecting the first instances with the first instance quantity from the pre-deployed migration microservice instances with the second instance quantity, the second instances can be prevented from performing data migration. And by adjusting the service ratio of the first instances and the second instances, most services are processed through the deployment containers corresponding to the second instances, so that stable services can be provided for normal services while reducing the impact of normal services on data migration. At the same time, by setting multiple first instances, the time required for migration can be further compressed. And the first instances integrate the service logic, so that customized migration of the migrated data can be realized. Therefore, compared with the related art, the embodiment of the present application can not only realize the customized migration of the migrated data but also improve the migration efficiency.

[0071] Exemplarily, refer to Figure 2As shown in the figure, taking the data to be migrated as bank statements and the normal business process as electronic transactions as an example, the bank statements are stored in the source database. The number of second instances is set to 4, and the migration microservice instances are respectively deployed in containers 1, 2, 3, and 4. The target database is the destination of the bank statement migration. Taking the number of first instances as 2 as an example, assuming that the migration microservice instances deployed in containers 1 and 2 are the first instances, the business proportion on containers 3 and 4 will increase. Correspondingly, the business proportion on containers 1 and 2 will decrease. At this time, for containers 1 and 2, when migrating bank statements, since the proportion of the business processing of electronic transactions is relatively low, therefore, the business processing of electronic transactions will not affect the data migration efficiency. Therefore, compared with the related technology, it can reduce the impact of normal business processing on data migration. At the same time, since the business proportion on containers 3 and 4 is increased, most of the business will run preferentially in containers 3 and 4, thereby reducing the impact of data migration on normal business, and making the business processing of the entire system relatively stable. At the same time, the data to be migrated is processed by containers 1 and 2 simultaneously, realizing multi-container asynchronous processing, and the migration efficiency is higher.

[0072] It can be understood that, referring to Figure 3 As shown in the figure, in some embodiments, step S103, according to the preset business logic, splitting the data to be migrated into data packets with the number of first instances, includes:

[0073] Determining the main table and the slave table in the data to be migrated according to the association relationship between the tables in the data to be migrated;

[0074] Splitting the main table into the first data sets with the number of the first instances;

[0075] According to each first data set, splitting the data of the slave table to obtain a plurality of second data sets corresponding one by one to the first data sets;

[0076] According to the business logic, integrating each first data set and the corresponding second data set respectively to obtain data packets with the number of the first instances.

[0077] It should be noted that the data of the slave table corresponds to the sub-items of one or more data items in the main table. For example, the main table is set with a time item and a task item, and the slave table is a record of the tracking situation of the progress of the time item and the task item.

[0078] It should be noted that the splitting of the main table is based on the total length of the data to be migrated. Since the main table and the slave table have a uniquely determined relationship, therefore, the length of each data packet can be determined based on their association relationship and the data volume of each table, and then the main table can be split according to the length of each data packet to obtain the first data sets with the number of the first instances, and then the second data sets can be determined according to the first data sets.

[0079] It should be noted that there can be multiple or one main table. For the main table, it can have a slave table or no slave table. In this regard, the embodiments of the present application do not limit the association relationship between the main table and the slave table, the number of main tables, or the number of slave tables under each main table. Those skilled in the art can determine according to actual business requirements. The first data set is a set of data records in the main table that are divided into one data packet, and the second data set is a set of data records of the slave table corresponding to the first data set. Taking the electronic transaction records in bank statements as an example, the main table records user information, and the slave table is the electronic transaction records of each user of different types; thus, the electronic transaction records of users can be jointly managed through the main table and the slave table.

[0080] It should be noted that the association relationship is used to distinguish whether each table records the same type of data. For multiple tables that record the same type of data, they can be divided into slave tables and main tables. The present application does not limit how to determine the main table and the slave table, and those skilled in the art can determine according to actual needs.

[0081] Exemplarily, referring to Figure 3 As shown, Table 1 is provided with Data Item 1, Data Item 2, and Data Item 3; Table 2 is provided with Data Item 1, Data Item 5, and Data Item 6; Table 3 is provided with Data Item 2, Data Item 7, and Data Item 8. That is, both Table 2 and Table 3 are expanded around Table 1 and record the same type of data. Therefore, it can be determined that Table 1 is the main table, and Table 2 and Table 3 are slave tables. And the data volume corresponding to each record in Table 1 in Table 2 and Table 3 can be obtained accordingly. Assuming that Table 1 is divided by Data Item 3 and three first data sets are obtained, namely Data Set 1, Data Set 2, and Data Set 3; then the corresponding data packets can be determined according to the relationship between the data records in Data Set 1, Data Set 2, and Data Set 3 and Table 1 and Table 2.

[0082] It can be understood that in some embodiments, step S103, according to a preset business logic, splitting the data to be migrated into data packets of the first instance number includes:

[0083] Obtaining the first data size of the data to be migrated;

[0084] According to the first data size and the first instance number, determining the second data size corresponding to each data packet; wherein, the difference between any two second data sizes meets a preset threshold;

[0085] Splitting the data to be migrated according to the second data size corresponding to each data packet to obtain multiple data packets.

[0086] It should be noted that the first data size represents the data length of the data to be migrated. The second data size represents the length of the data packet, and the data packets can be of the same size or different sizes. For example, the second data size can be set to 10M for all, or one data packet is 10M and another data packet is 11M.

[0087] It should be noted that the preset threshold is used to make the time taken for each deployment container to complete data packet sending similar, such as not exceeding one minute. It should be noted that the time taken for data packet sending can be determined based on hardware parameters. When the hardware parameters are the same, it is default that the time taken for each deployment container to send data packets of the same length is equal, and at this time, the data to be migrated is equally divided. When the hardware parameters are different, the hardware parameters can be classified, and the second data size of the data packets to be sent by each deployment container can be determined according to the level corresponding to each deployment container. The hardware parameters can be the CPU processing power or the memory size, etc. In this regard, the embodiments of the present application do not limit the setting of the preset threshold, and those skilled in the art can selectively set it based on transmission efficiency, etc.

[0088] It can be understood that in step S101, selecting the first instance of the first instance number from the migration microservice instances of the second instance number pre-deployed includes:

[0089] Sorting the migration microservice instances of the second instance number according to the network parameters of the deployed deployment containers;

[0090] Selecting the first instance of the first instance number from the sorted migration microservice instances of the second instance number.

[0091] It should be noted that the network parameters are parameters related to network transmission, such as IP address, MAC address, transmission bandwidth, etc. The sorting can be in descending order or ascending order, and its sorting is set based on network parameters. For example, if the network parameters are set to IP address or MAC address, etc., ascending order can be used. If the network parameters are set to transmission bandwidth, descending order can be used. It should be noted that the migration microservice instances can be sorted based on one network parameter or multiple network parameters. For example, if there are two network parameters, IP address and transmission bandwidth, sorting can be performed respectively based on the IP address and transmission bandwidth, and weight calculation can be performed based on the sorting results of the two sorts to determine the final sorting. The embodiments of the present application do not limit the sorting method, and those skilled in the art can selectively sort according to actual needs.

[0092] Exemplarily, taking the IP address as the network and the sorting being in descending order as an example, after containers 1, 2, and 3 are sorted in descending order according to the IP address, they are sorted as container 2, container 3, and container 1 in sequence. Assuming the first instance number is 2, the migration microservice instances on containers 2 and 3 are the first instances, and container 1 is the second instance.

[0093] It is understood that selecting the first instances with the first instance quantity from the migrated microservice instances with the pre-deployed second instance quantity in step S101 further includes:

[0094] Assigning identification codes to each of the first instances respectively according to the arrangement order of each first instance;

[0095] Correspondingly, step S104, sending the data packets with the first instance quantity to the target database one by one through the corresponding first instances, includes:

[0096] Associating each identification code with each data packet one by one;

[0097] Sending each data packet to the target database one by one through the first instance corresponding to the associated identification code respectively.

[0098] It should be noted that by setting the identification code, the data packets are sent to the target database one by one by the corresponding first instances according to the identification code. At this time, the source database does not need to care about the specific deployment container, and only needs to determine that there is a uniquely corresponding identification code, thereby improving the processing efficiency.

[0099] In some embodiments, the data migration method further includes:

[0100] When there are faulty first instances, reordering the non-faulty first instances;

[0101] Reassigning identification codes to each of the re-ordered first instances;

[0102] Determining the distribution codes corresponding to the identification codes that have not been re-assigned according to the preset conversion rules, wherein the distribution codes are in one-to-one correspondence with the assigned identification codes;

[0103] Sending the data packets corresponding to the distribution codes to the target database one by one through the corresponding first instances.

[0104] It should be noted that the fault can be a fault such as downtime or link break that causes the first instance to be unable to execute normally. At this time, by re-setting the distribution code, each data packet can be sent again with a uniquely determined deployment container, reducing the situation of out-of-order or data packet unable to be sent.

[0105] It should be noted that in some embodiments, after the distribution code is re-determined, the identification code associated with the data packet can be directly modified. In other embodiments, there is an independent conversion module. When it is determined that the data packet is the one with the identification code that has not been re-assigned, after the conversion module determines the distribution code, it is sent out based on the distribution code.

[0106] It should be noted that it can be that the first migration microservice example actively requests the data packet corresponding to the identification code and the data packet corresponding to the distribution code allocated to it from the source database. In this case, after determining the distribution code, it will notify the unredistributed identification code associated with the data packet to be extracted by the first instance corresponding to the distribution code. In some other embodiments, it can be that the source database actively sends the data packet to the first migration microservice example. In this case, the received first migration microservice example will be determined based on the distribution code and the identification code allocated at the start of data migration.

[0107] It should be noted that the conversion rule is used to record the mapping relationship between the k data packets that have not been redistributed and the first k migration microservice examples. The distribution code is the identification code redistributed to the first migration microservice example determined according to this mapping relationship. Therefore, the distribution code and the redistributed identification code are in one-to-one correspondence.

[0108] It should be noted that for the first instance corresponding to the redistributed identification code, it sends the data packet corresponding to the redistributed identification code to the target database.

[0109] Exemplarily, referring to Figure 4 As shown, assume there are M first instances (instance 1, instance 2,..., instance n respectively), and the allocation codes are 1, 2, 3,..., m respectively. The allocation codes associated with the M data packets (data packet 1, data packet 2,..., data packet m respectively) are 1, 2, 3,..., m one by one. Assume that instance 2 with an allocation code of 2 and instance 4 with an allocation code of 4 fail, such as crashing. Then the remaining M - 2 first instances are reordered, that is, the allocation codes corresponding to instance 1, instance 3, instance 5, instance 6,..., instance n are 1, 2, 3,..., m - 2 respectively. At this time, for data packet m with an allocation code of m, since the allocation code increases sequentially, the conversion rule sets the mapping relationship as the remainder obtained by dividing the identification code of the unredistributed data packet by the number of non-failed first migration microservices. Then for data packet m - 1, the identification code is m - 1; its distribution code is (m - 1) % (M - 2). Specifically, taking M as 10 as an example, the allocation code corresponding to data packet 9 is 9 % 8 = 1. Therefore, data packet 9 is sent to the first instance with an allocation code of 1.

[0110] In some embodiments, the method for data migration further includes:

[0111] During the migration of the data to be migrated, obtain the updated data to be migrated according to a preset time period;

[0112] Perform data migration on each piece of updated data respectively.

[0113] It should be noted that during each data migration process, there may be continuous data updates. By periodically obtaining updated data and migrating and synchronizing it one by one, the data can be processed in batches, reducing the impact of the migrated data on normal business operations and improving the migration efficiency.

[0114] It should be noted that when there is updated data, the first microservice migration instance will initiate an update task. The update task migrates each piece of updated data. The update task can correspond one-to-one with the updated data, or the update task can be started after each data migration. In this regard, the embodiments of the present application do not make any restrictions, and those skilled in the art can set it according to actual needs.

[0115] Exemplarily, with reference to Figures 1 to 4 as shown, the method for data migration in the embodiments of the present application is described as follows:

[0116] With reference to Figure 1As shown, taking the first instance quantity set to 3 as an example, containers 1, 2, 3, and 4 are arranged according to their IP addresses, and the obtained order is container 3, container 1, container 2, and container 4 in sequence. At this time, referring to step S101, the migration microservice instances correspondingly deployed in containers 3, 1, and 2 are all first instances. Then referring to step S102, the business weights of containers 3, 1, and 2 are reduced, and the business weight of container 4 is increased. Then the data migration process is started. During the data migration process, referring to step S103, according to the preset business logic, the data to be migrated in the source database is divided into 3 equal data packets. When splitting, an index is selected from the main table of the data to be migrated, and the data records are split based on this index. After the splitting is completed, the sequential numbers of the data packets are used as identification codes one by one to establish associations according to the splitting order of the data packets. Similarly, containers 3, 1, and 2 are given sequential numbers as identification codes according to their arrangement order. At this time, there is a one-to-one correspondence between the data packets and containers 3, 1, and 2. Referring to step S104, the first instance on container 3 sends the first data packet to the target database, the first instance on container 1 sends the second data packet to the target database, and the first instance on container 2 sends the second data packet to the target database. When container 3 fails, after reordering, the container order is: container 1, container 2. At this time, the identification code of container 1 is 1, and the identification code of container 2 is 2. For the third data packet, its corresponding identification code is the unassigned identification code. According to the preset conversion rule, the assigned code corresponding to the third data packet is 3 % 2, which is 1. Therefore, the third data packet is sent to the target server by the first instance on container 1. Before the first, second, and third data packets are sent, there is data update. At this time, updated data is periodically generated. After the first, second, and third data packets are sent, the update task is started, and data migration processing is performed one by one according to the generation time of the updated data to achieve synchronous update of the data.

[0117] Please refer to Figure 5 , this embodiment of the present application also provides a data migration device, which can implement the above data migration method. The device includes:

[0118] A selection module 100, configured to obtain the configured first instance quantity, and select the first instance quantity of first instances from the migration microservice instances of the pre-deployed second instance quantity; wherein, the second instance quantity is greater than the first instance quantity;

[0119] An adjustment module 200, configured to adjust the business weights of the deployment containers corresponding to the first instances according to the preset proportional adjustment rule, and increase the business weights of the deployment containers corresponding to the second instances that are not selected among the migration microservice instances of the second instance quantity;

[0120] The splitting module 300 is configured to split the data to be migrated into data packets with a first instance quantity according to a preset service logic, and the data packets with the first instance quantity correspond to the first instances with the first instance quantity one by one;

[0121] The migration module 400 is configured to send the data packets with the first instance quantity to the target database one by one through the corresponding first instances.

[0122] It should be noted that, in some embodiments, referring to Figure 6 As shown, the migration module 400 includes an allocation module 410 and a synchronization module 420. The allocation module 410 is configured to determine the identification codes of each available first instance, and determine the data packets to be synchronized according to the identification codes. The synchronization module 420 is configured to extract the data packets corresponding to the identification codes to the corresponding first instances, and then send the corresponding data packets to the target database through the first instances.

[0123] The specific implementation manner of this data migration device is basically the same as the specific embodiments of the above data migration method, and will not be elaborated here.

[0124] It should be noted that since the migration logic itself (i.e., the service logic) is implemented in the service system, the developed class libraries in the service system can be reused to improve the efficiency of requirement development. The design of splitting the overall data into data packets effectively isolates the migrated data between multiple instances, preventing data from being migrated repeatedly and causing anomalies. And the data packets are evenly distributed to multiple running data migration task instances as reasonably as possible to improve the migration efficiency.

[0125] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the above data migration method is implemented. This electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0126] Please refer to Figure 7 , Figure 7 which schematically shows the hardware structure of an electronic device in another embodiment. The electronic device includes:

[0127] A processor 501, which can be implemented by using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present application;

[0128] The memory 502 can be implemented in the form of a Read-Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 502 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 502 and are called by the processor 501 to execute the data migration method of the embodiments of this application;

[0129] The input / output interface 503 is used to implement information input and output;

[0130] The communication interface 504 is used to implement communication and interaction between this device and other devices. It can achieve communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0131] The bus 505 transmits information between various components of the device (such as the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0132] Among them, the processor 501, the memory 502, the input / output interface 503, and the communication interface 504 achieve communication connections with each other inside the device through the bus 505.

[0133] The embodiments of this application also provide a storage medium. The storage medium is a computer-readable storage medium. This storage medium stores a computer program, and when the computer program is executed by a processor, it implements the above-mentioned data migration method.

[0134] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include high-speed random access memory, and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory optionally includes a memory remotely set relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and their combinations.

[0135] The method for data migration, the device for data migration, the electronic device, and the storage medium provided by the embodiments of the present application select the first number of first instances from the second number of migration microservice instances deployed in advance, so that the second instances do not need to perform data migration. By adjusting the business proportion of the first instances and the second instances, most of the business is processed through the deployment containers corresponding to the second instances, so that stable services can be provided for normal business while reducing the impact of normal business on data migration. At the same time, by setting multiple first instances, the time required for migration can be further compressed. The first instances integrate business logic, so that customized migration of the migrated data can be achieved. Therefore, compared with the related art, the embodiments of the present application can not only achieve customized migration of the migrated data but also improve the migration efficiency.

[0136] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0137] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or combine some steps, or different steps.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices can be implemented as software, firmware, hardware, and their appropriate combinations.

[0140] In the description of the present application and the above-mentioned drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0141] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally means that the associated objects before and after are in an "or" relationship. "At least one (one) of the following" or its similar expression means any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0142] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the above-mentioned division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0143] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, in each embodiment of the present application, the functional units can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: various media that can store programs, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0146] The preferred embodiments of the embodiments of the present application have been described above with reference to the accompanying drawings, and thus do not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements, and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.

Claims

1. A method for data migration, characterized in that, The method includes: Obtaining the configured number of first instances, and selecting the first instances of the number of first instances from the pre-deployed migration microservice instances of the number of second instances; wherein, the number of second instances is greater than the number of first instances; According to a preset proportion adjustment rule, reducing the business proportion of the deployment containers corresponding to the first instances, and increasing the business proportion of the deployment containers corresponding to the second instances that are not selected among the migration microservice instances of the number of second instances; According to a preset business logic, splitting the data to be migrated into data packets of the number of first instances, and the data packets of the number of first instances correspond one-to-one with the first instances of the number of first instances; Sending the data packets of the number of first instances to the target database one by one through the corresponding first instances; The splitting the data to be migrated into data packets of the number of first instances according to a preset business logic includes: Determining the main table and slave tables in the data to be migrated according to the association relationship between the tables in the data to be migrated; splitting the main table into the first data sets of the number of first instances; splitting the slave tables according to each of the first data sets to obtain a plurality of second data sets corresponding one-to-one with the first data sets; according to the business logic, respectively integrating each of the first data sets and the corresponding second data sets to obtain data packets of the number of first instances; The selecting the first instances of the number of first instances from the pre-deployed migration microservice instances of the number of second instances includes: Sorting the migration microservice instances of the number of second instances according to the network parameters of the deployed deployment containers; selecting the first instances of the first number of instances from the sorted migration microservice instances of the number of second instances.

2. The method for data migration according to claim 1, wherein The splitting the data to be migrated into data packets of the number of first instances according to a preset business logic includes: Obtaining the first data size of the data to be migrated; According to the first data size and the number of first instances, determining the second data size corresponding to each of the data packets; wherein, the difference between any two of the second data sizes meets a preset threshold; Splitting the data to be migrated according to the second data size corresponding to each of the data packets to obtain a plurality of the data packets.

3. The method according to claim 1, wherein The selecting the first instances of the number of first instances from the pre-deployed migration microservice instances of the number of second instances further includes: According to the arrangement order of each of the first instances, respectively assigning identification codes to each of the first instances; Correspondingly, the sending the data packets of the number of first instances to the target database one by one through the corresponding first instances includes: Associating each of the identification codes with each of the data packets one-to-one; Respectively sending each of the data packets to the target database one by one through the first instance corresponding to the associated identification code.

4. The method according to claim 3, characterized in that, The method further includes: When there are faulty first instances, re-sorting the non-faulty first instances; Re-assigning the identification codes to each of the re-sorted first instances; According to a preset conversion rule, determining the distribution codes corresponding to the identification codes that are not re-assigned, wherein the distribution codes correspond one-to-one with the assigned identification codes; Send the data packets corresponding to the distribution code to the target database one by one through the corresponding first instance.

5. The method for data migration according to any one of claims 1 to 4, characterized in that The method further includes: During the migration of the data to be migrated, obtain the updated data to be migrated according to a preset time period; Perform data migration on each piece of the updated data respectively.

6. A device for data migration, characterized in that, The apparatus includes: A selection module, configured to obtain the configured number of first instances, and select the first instances of the number of first instances from the migration microservice instances of the pre-deployed number of second instances; wherein, the number of second instances is greater than the number of first instances; An adjustment module, configured to adjust the service ratio of the deployment containers corresponding to the first instances according to a preset ratio adjustment rule, and increase the service ratio of the deployment containers corresponding to the second instances that are not selected among the migration microservice instances of the number of second instances; A splitting module, configured to split the data to be migrated into data packets of the number of first instances according to a preset business logic, and the data packets of the number of first instances correspond one by one to the first instances of the number of first instances; A migration module, configured to send the data packets of the number of first instances to the target database one by one through the corresponding first instances; The splitting of the data to be migrated into data packets of the number of first instances according to a preset business logic includes: Determine the master table and the slave tables in the data to be migrated according to the association relationships of the tables in the data to be migrated; split the master table into the first data sets of the number of first instances; perform data splitting on the slave tables according to the first data sets to obtain a plurality of second data sets corresponding one by one to the first data sets; integrate each of the first data sets and the corresponding second data sets according to the business logic to obtain data packets of the number of first instances; The selection of the first instances of the number of first instances from the migration microservice instances of the pre-deployed number of second instances includes: Sort the migration microservice instances of the number of second instances according to the network parameters of the deployed deployment containers; select the first instances of the first number of instances from the sorted migration microservice instances of the number of second instances.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the data migration method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data migration method according to any one of claims 1 to 5.

Images