Data migration method and device, equipment, storage medium and program product

Abstract

The invention discloses a data migration method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing.The data migration method includes the steps that according to a preset access condition, access is conducted from a data source library to obtain source table row data; according to a preset comparison condition, comparing the source table row data with preset target table row data to obtain data to be newly added and data to be deleted; and inserting the to-be-newly added data into the target database, and deleting the to-be-deleted data. According to the method, condition-based filtering and access of the source table data are achieved through the preset access condition, the source table data are mapped into the target format during access, data difference comparison of different databases is achieved through the comparison fields, high-performance access of the source databases is achieved through the stepping fields, the heterogeneous data migration efficiency between the different databases is effectively improved, and the heterogeneous data migration efficiency is improved. And system resources are saved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, apparatus, device, storage medium, and program product. Background Art

[0002] Currently, incrementally migrating heterogeneous data between different databases typically involves streaming data using technologies or middleware such as ETL (Extract, Transform, Load), Kafka (a distributed stream processing platform), and Spark (a distributed computing framework). Business data read from the data stream is then hard-coded into the target format for storage. While this approach can also achieve data extraction, comparison, and migration, it utilizes multiple middleware implementations and hard-coded data mappings. This leads to complex technology stacks, high resource consumption, hard-coded data mappings, and poor timeliness in ETL production. It also requires significant investment in manpower and resources, and requires customized development for different projects, making it difficult to universally apply. Summary of the Invention

[0003] The main purpose of this application is to provide a data migration method, device, equipment, storage medium and program product, aiming to solve the technical problem of how to improve the efficiency of heterogeneous data migration while reducing the migration cost.

[0004] To achieve the above objectives, this application proposes a data migration method, which includes:

[0005] According to the preset data retrieval conditions, the source table row data is retrieved from the data source library;

[0006] According to the preset comparison conditions, the source table row data and the preset target table row data are compared to obtain the data to be added and the data to be deleted;

[0007] Insert the data to be added into the target database, and delete the data to be deleted.

[0008] In one embodiment, the preset data retrieval condition includes at least a step field and a data retrieval filter condition. The step of retrieving data from a data source library to obtain source table row data according to the preset data retrieval condition includes:

[0009] According to the step field, the data source library is fetched page by page to obtain the initial source table row data;

[0010] When an instruction to execute a data retrieval filter condition is identified, the initial source table data is filtered according to the data retrieval filter condition to obtain source table row data that meets the data retrieval filter condition;

[0011] According to the fetch field mapping, the filtered source table row data is format mapped to obtain source table row data in a target format.

[0012] In one embodiment, the step of fetching data page by page from the data source library according to the step field to obtain the initial source table row data includes:

[0013] Counting the total amount of data in the data source library that meets the data retrieval filtering conditions to determine the paging size;

[0014] Generate a paging query condition based on the value range of the step field and the paging size;

[0015] According to the paging query conditions, data is extracted from the data source library in sequence to obtain initial source table data.

[0016] In one embodiment, after the step of fetching data from the data source library page by page according to the step field to obtain initial source table row data, the step further includes:

[0017] When no instruction for executing the data retrieval filtering condition is identified, a corresponding empty array is configured for the source table row data as a filtering result identifier.

[0018] In one embodiment, the step of performing format mapping on the filtered source table row data according to the access field mapping to obtain the source table row data in the target format includes:

[0019] Determine whether the fields of the filtered source table row data have changed based on business needs;

[0020] If the fields of the filtered source table row data do not change, the fields of the filtered source table row data are matched one-to-one with the fields of the preset target table row data;

[0021] If the fields of the filtered source table row data change, the keys and values in the fields of the filtered source table row data are mapped to the target format according to the preset field mapping rules and conditional judgment syntax to obtain the source table row data in the target format.

[0022] In one embodiment, the step of comparing the source table row data with the preset target table row data according to a preset comparison condition to obtain the data to be added and the data to be deleted includes:

[0023] Extracting the key field values of the source table row data and the preset key field values of the target table row data;

[0024] Determine whether the key field value of the source table row data is the same as the key field value of the preset target table row data;

[0025] If the key field value of the source table row data is different from the key field value of the preset target table row data, the source table row data is marked as data to be added;

[0026] If the key field value of the source table row data is the same as the key field value of the preset target table row data, the source table row data is marked as existing data, and the existing data is confirmed to be data to be deleted.

[0027] In addition, to achieve the above-mentioned purpose, the present application also proposes a data migration device, which includes:

[0028] The data retrieval module is used to retrieve source table row data from the data source library according to the preset data retrieval conditions;

[0029] A comparison module is used to compare the source table row data with the preset target table row data according to preset comparison conditions to obtain data to be added and data to be deleted;

[0030] The processing module is used to insert the data to be added into the target database and delete the data to be deleted.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a data migration device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data migration method described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the data migration method described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the data migration method described above.

[0034] The embodiment of the present application provides a data migration method, apparatus, device, storage medium and program product, the method comprising: obtaining source table row data from a data source library according to preset data retrieval conditions; comparing the source table row data with preset target table row data according to preset comparison conditions to obtain data to be added and data to be deleted; inserting the data to be added into the target database, and deleting the data to be deleted. The method implements conditional filtering and data retrieval of source table data through preset data retrieval conditions, maps source table data to a target format when retrieval, compares data differences between different libraries through comparison fields, and implements high-performance data retrieval of the source library through stepping fields, effectively improving the efficiency of heterogeneous data migration between different databases and saving system resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 A flowchart of the first embodiment of the data migration method of this application is provided;

[0038] Figure 2 This is a schematic diagram of data retrieval and filtering provided in Example 1 of this application;

[0039] Figure 3 A schematic diagram of field mapping of source table row data provided in Example 1 of the present application;

[0040] Figure 4 A schematic diagram of the comparison field configuration provided in Example 1 of this application;

[0041] Figure 5 A flowchart of the second embodiment of the data migration method of this application is provided;

[0042] Figure 6 A brief flowchart of the first embodiment of the data migration method of this application is provided;

[0043] Figure 7 A data difference comparison flowchart provided for the data migration method of this application;

[0044] Figure 8 This is a schematic diagram of the module structure of the data migration device according to an embodiment of the present application;

[0045] Figure 9 Schematic diagram of the device structure of the hardware operating environment involved in the data migration method in the embodiment of the present application. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The main solution of the embodiment of the present application is: according to the preset data retrieval conditions, data is retrieved from the data source library to obtain source table row data; according to the preset comparison conditions, the source table row data is compared with the preset target table row data to obtain the data to be added and the data to be deleted; the data to be added is inserted into the target database, and the data to be deleted is deleted.

[0049] In this embodiment, for ease of description, the following description is made with a personal computer as the execution subject.

[0050] Currently, incrementally migrating heterogeneous data between different databases typically involves streaming data using technologies or middleware such as ETL, Kafka, and Spark. Business data read from the data stream is then hard-coded into the target format for storage. While this approach can achieve data extraction, comparison, and migration, it utilizes multiple middleware implementations and hard-coded data mappings. This leads to complex technology stacks, high resource consumption, hard-coded data mappings, and poor timeliness in ETL production. This approach also requires significant investment in manpower and resources, and requires customized development for different projects, making it difficult to universally apply.

[0051] This application provides a solution that extracts source table data page by page based on parameters such as step fields and paging size, which can avoid loading large amounts of data at one time and reduce the system burden; accurately identifies the data that needs to be processed through detailed comparison of key fields and field values, which can reduce unnecessary data transmission and processing; supports complex mapping rules, including direct mapping, function calculation and expression conversion, to ensure that data can be seamlessly migrated to the target database; and only performs insert or update operations on changed data, avoiding the performance overhead caused by full synchronization, thereby improving overall migration efficiency.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions. The following uses a personal computer as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides a data migration method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the data migration method of this application.

[0054] In this embodiment, the data migration method includes steps S10 to S30:

[0055] Step S10: Get source table row data from the data source library according to the preset data retrieval conditions;

[0056] It's important to note that source table row data refers to specific records or rows extracted from the source repository (i.e., the original database or system). These rows contain specific information, typically associated with a business entity, such as user information, transaction records, or device status.

[0057] In addition, it should be noted that, in this embodiment, the data acquisition conditions at least include a data acquisition target table, a data acquisition field mapping, a filtering condition, a step field, and the like.

[0058] For data retrieval condition configuration, the source table uses the format schema.table_name, explicitly indicating the database table from which data is to be read. The data retrieval field mapping configuration is a JSON string in map format, which details the identifier of each field in the source table and the data mapping rules. The data retrieval filter condition limits the data range to be extracted from the source table, enabling on-demand data retrieval. The step field configuration significantly improves overall program performance through paging.

[0059] It is understandable that since the data source library usually contains a large amount of redundant or irrelevant data, direct full extraction will lead to resource waste and performance bottlenecks, so step S10 is performed to reduce unnecessary data loading and transmission overhead, thereby improving data retrieval efficiency and overall system performance.

[0060] In a feasible implementation, step S10 may include steps S11 to S13:

[0061] Step S11, fetching data page by page from the data source library according to the step field to obtain initial source table row data;

[0062] It should be noted that the step field refers to a key field used for paging and processing data segment by segment during the data extraction process.

[0063] It is understandable that for scenarios with large amounts of source table data, simply using page numbers for paging can lead to inefficient data retrieval queries, and occupying the resource library's CPU and memory resources. Therefore, step S11 is executed, and the step field configuration item supports selecting a numeric or string field with high data differentiation, low repetition rate, and usable indexes. Compared to conventional paging by page number, the conditional paging method implemented using step fields can improve performance by up to 100 times in the case of 10 million data in a single table.

[0064] In another feasible embodiment, step S11 may further include steps A11 to A13:

[0065] Step A11, counting the total amount of data in the data source library that meets the preset data retrieval conditions to determine the paging size;

[0066] Specifically, SQL statements are used to query the data source library to count the total number of data that meet the preset filtering conditions. Then, based on the statistical results and system performance requirements, the amount of data extracted per page (page size) is determined.

[0067] For example, if the total data volume is 100,000 and a single query recommends no more than 1,000 records, the paging size is set to 1,000.

[0068] Assume that the total amount of data that meets the data retrieval conditions in the source table users in the data source library is total_records=15300, and the paging size is set to page_size=1000, then the total number of pages is ceil(15300 / 1000)=16 pages.

[0069] Through the above steps, the total amount of data to be processed is determined, and the paging size is determined based on this to balance performance and resource consumption.

[0070] Step A12: generating a paging query condition based on the value range of the step field and the paging size;

[0071] Specifically, first, determine the value range of the progress field. For example, if the progress field is id (primary key), its maximum and minimum values need to be obtained, where the maximum and minimum values may be min_id=1 and max_id=15300.

[0072] Then, according to the paging size and the value range of the progress field, the starting value and the ending value of each page are calculated, and the paging query conditions are generated based on the starting value and the ending value of each page.

[0073] For example, if the step field is id and the paging size is 1000, the paging condition generation logic is as follows:

[0074] Starting value: start_id = min_id + (page_number - 1) * page_size;

[0075] End value: end_id = start_id + page_size - 1;

[0076] For page n, the query condition is: WHERE id BETWEEN:start_id AND:end_id

[0077] AND<preset data retrieval condition>.

[0078] By stepping the value range of the field and the paging size, paging query conditions are generated to extract data page by page, ensuring the continuity and integrity of data paging.

[0079] Step A13: extract data from the data source library in sequence according to the paging query conditions to obtain initial source table data.

[0080] Specifically, based on the generated paging query conditions, data from the source table is extracted page by page in the data source library to obtain the initial source table data. If the data on a page does not fit within the paging size (such as the last page), the remaining data is directly extracted. If the paging conditions are incorrect (such as the stepping field is not unique or contains null values), exception handling logic must be added.

[0081] For example, assuming the paging size is 1000 and the total number of pages is 16, then data with IDs 1-1000 are extracted from page 1, totaling 1000 records. Data with IDs 15001-15300 are extracted from page 16, totaling 300 records. After merging, 15,300 records that meet all the requirements are obtained.

[0082] To summarize, by executing paging queries page by page, data can be gradually extracted to avoid performance issues caused by loading a large amount of data at one time.

[0083] Step S12: When an instruction to execute the data retrieval filtering condition is identified, the initial source table data is filtered according to the data retrieval filtering condition to obtain source table row data that meets the data retrieval filtering condition;

[0084] It should be noted that the filter condition is configured as a JSON string in array format, which configures the type of filter condition to be used to filter the source table data during the data retrieval process from the source table, and ultimately removes the source table data that meets the filter condition requirements.

[0085] Specifically, first, select appropriate filtering conditions based on business needs. When the system recognizes the instruction to execute the filtering condition, the source table data is filtered according to the filtering condition, and finally the source table data that meets the filtering condition is obtained. Among them, the filtering conditions include but are not limited to time range filtering, for example, only selecting data from the last three months for migration or analysis. Numerical range filtering, for example, filtering all records with sales greater than a certain amount. Keyword matching filtering, such as searching for relevant product descriptions or customer reviews by keywords. Status identification filtering, filtering data according to the status of the record (for example, "processed", "unprocessed"). Complex logical expressions, combining the values of multiple fields using logical operators (AND, OR, NOT, etc.) to build more complex filtering conditions.

[0086] Please refer to Figure 2 , Figure 2 This is a diagram of data filtration, where: Figure 2 "filterField n" = "valuel n" indicates the filtering condition. filterField n is the name of a field in the source table, and valuel n is the specific value that needs to be matched in the corresponding field.

[0087] For example, consider a sales data table with the fields sales, state, and region. The JSON configuration for these fields can be interpreted as follows: filterField1 corresponds to sales, with a value of value1; filterField2 corresponds to state, with a value of value2; and filterField3 corresponds to region, with a value of value3.

[0088] If value1 is set to 1000, value2 to "Processed", and value3 to "East China" or "South China", the filter conditions can be expressed as: Sales > 1000; Status = "Processed"; Region IN ("East China", "South China").

[0089] Through the above steps, we define filter conditions using structured JSON configuration, combined with logical operators and field type adaptation, to achieve precise filtering of the original source table data. This flexible and scalable configuration approach is suitable for data filtering needs in complex business scenarios. It also ensures processing efficiency and accuracy through reasonable logical combination and performance optimization.

[0090] Step S13 : performing format mapping on the filtered source table row data according to the fetch field mapping to obtain source table row data in a target format.

[0091] It is understandable that since the field structures of the source table and the target table may differ (such as field name, data type, length limit, etc.), step S13 is executed to convert the filtered source table row data obtained in step S12 into the target format according to the acquired field mapping configuration.

[0092] In a feasible embodiment, step S13 may include steps B11 to B13:

[0093] Step B11, determining whether the fields of the filtered source table row data have changed according to business requirements;

[0094] Specifically, determine whether the status of the fields used to format the filtered source table row data has changed, including but not limited to checking whether the field is empty, whether the data type is normal, and whether the field value meets business logic requirements (for example, whether a certain value is within the allowable range). Determine whether the field needs to be changed based on business needs. For example, if business rules require all new users to set an initial score, then the corresponding field value needs to be checked and supplemented.

[0095] Through the above steps, determine whether the filtered source table row data needs to be further converted or adjusted to meet the requirements of the target table.

[0096] Step B12: if the fields of the filtered source table row data do not change, the fields of the filtered source table row data are matched one-to-one with the fields of the preset target table row data;

[0097] Specifically, when the fields of the to-be-added data do not require additional processing, the fields are directly matched one-to-one with the corresponding fields in the target table.

[0098] Step B13: If the fields of the filtered source table row data change, the keys and values in the fields of the filtered source table row data are mapped to the target format according to the preset field mapping rules and conditional judgment syntax to obtain the source table row data in the target format.

[0099] It should be noted that the preset field mapping rule is the basic configuration for implementing heterogeneous data mapping. It configures the method of converting field A in the source table into field B in the target table using certain conversion rules.

[0100] Specifically, when it is detected that the fields of the filtered source table row data have changed (such as data type conversion, formatting requirements, etc.), the format mapping is performed in the source table using functions and conditional judgment (case-when) syntax according to the preset field mapping rules. Please refer to Figure 3 , Figure 3 This is a diagram showing the field mapping of the source table row data. Figure 3In the example, source field field1 is mapped to target field fieldA; source field field2 is mapped to target field fieldB; source field case field3 when'val3.1'then'1'else'0'end is mapped to target field fieldC. Here, the CASE-WHEN syntax is used to conditionally judge field3. If the value of field3 is 'val3.1', the value of fieldC is '1'. Otherwise, the value of fieldC is '0'. Source field concat(field4, field5) is mapped to target field fieldD. Here, the CONCAT function is used to concatenate the values of field4 and field5, and the concatenated result is used as the value of fieldD. Source field sum(field5, field6) is mapped to target field fieldE. Here, the SUM function is used to sum the values of field5 and field6, and the sum is used as the value of fieldE.

[0101] The above steps can flexibly handle various field mapping requirements and ensure that data is accurately converted and saved in the target database during the migration process.

[0102] Step S20: comparing the source table row data with the preset target table row data according to preset comparison conditions to obtain data to be added and data to be deleted;

[0103] It should be noted that, in this embodiment, the configuration of the comparison condition indicates whether the key field values corresponding to the source table row data and the target table row data are the same. When all key field values are the same, the source table row data and the target table row data are considered to be the same. Figure 4 , Figure 4 This is a diagram for configuring comparison fields. The comparison condition uses the comparison field to compare the source table row data with the target table row count. The comparison field is configured as a JSON string in array format, which contains several table field names. It is used to sequentially read the field values of the specified columns from the source and target tables and compare whether the field values are the same.

[0104] It is understandable that, because the preset comparison conditions allow the system to quickly locate differential data through key fields (such as primary keys, timestamps, etc.) and perform precise operations only on these differential data, executing step S20 eliminates the need to compare data between the source and target databases when performing full data synchronization. When implementing incremental data synchronization, the data in the two databases is compared line by line to determine the newly added, unchanged, and deleted data, avoiding the need to perform full database deletion and addition operations on the target database.

[0105] In a feasible embodiment, step S20 may include steps S21 to S24:

[0106] Step S21, extracting the key field values of the source table row data and the key field values of the preset target table row data;

[0107] It should be noted that key fields usually refer to fields that uniquely identify data rows (such as primary keys and unique index fields), for example: user_id in the user table; order_id in the order table; timestamp fields (such as last_modified_time) are used to determine data updates.

[0108] Step S22, determining whether the key field value of the source table row data is the same as the key field value of the preset target table row data;

[0109] Specifically, the unified key field values in the source table row data and the target table row data are compared to determine whether the data exists or needs to be updated. For example, if the key field value in the source table is id=101 and the key field value in the target table is id=101, then the key field values are the same.

[0110] Step S23: If the key field value of the source table row data is different from the key field value of the preset target table row data, the source table row data is marked as data to be added;

[0111] Specifically, if the comparison result indicates that the key field values of the source table row data are inconsistent with the key field values of the target table row data, the source table row data is marked as pending addition data. After all data is compared, the target table is checked for rows of data that do not exist in the source table but do exist in the target table and deleted. Subsequently, the data marked as pending addition is inserted into the target table, thus achieving incremental synchronization of single-table data across different databases.

[0112] Step S24: If the key field value of the source table row data is the same as the key field value of the preset target table row data, the source table row data is marked as existing data, and the existing data is confirmed to be data to be deleted.

[0113] Specifically, if the comparison result shows that the key field values of the source table row data are the same as the key field values of the target table row data, only the source table row data is marked as existing data, and the existing data is confirmed as data to be deleted. In this regard, direct data manipulation is not performed during the data comparison phase to avoid direct modification of source or target table data during the marking phase.

[0114] Through the above steps, the data of the two databases are compared row by row according to the preset comparison conditions to obtain the newly added, unchanged, and deleted data, avoiding the need to perform full database deletion and full database addition operations on the target database.

[0115] Step S30: insert the data to be added into the target database, and delete the data to be deleted.

[0116] Specifically, after all data comparisons are completed, the target table is checked for data that does not exist in the source table but exists in the target table, that is, the data to be deleted is deleted, and then the data to be added is inserted into the target table, realizing incremental synchronization of single-table data in different databases.

[0117] Through the above-mentioned embodiment method, according to the preset data retrieval conditions, data is retrieved from the data source library to obtain source table row data; according to the preset comparison conditions, the source table row data and the preset target table row data are compared to obtain the data to be added; the data to be added is converted according to the preset field mapping rules and saved in the target database. This method uses the preset data retrieval conditions for paging extraction, and combines accurate data comparison and flexible field mapping configuration to achieve a single-table data migration solution based on configuration, the simplest technology stack, high performance, and reusability. It can maximize resource savings and reduce customized development, providing theoretical support and implementation solutions for the rapid implementation of single-table data migration.

[0118] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 5 After step S11, the data migration method further includes step S110:

[0119] Step S110 : When no instruction for executing the data retrieval filtering condition is identified, a corresponding empty array is configured for the source table row data as a filtering result identifier.

[0120] Understandably, if the system doesn't detect a filtering condition or instruction, simply skipping filtering may result in subsequent processes being unable to determine whether the data has been filtered, thereby affecting the accuracy of data processing. Therefore, in this embodiment, by configuring an empty array as a filter result identifier for unfiltered source table rows, it is possible to clearly record that the data has not undergone any filtering operation, facilitating subsequent processing and tracking.

[0121] Specifically, the system checks for filtering conditions or instructions. If no filtering instructions are identified, the "Filter Result" field of each source table row is set to an empty array. An empty array indicates "no filtering," meaning the data remains in its original state without being filtered according to any conditions. The source table row data is then output with the filter result indicator (empty array) indicated.

[0122] The above embodiment can effectively implement the allocation of empty arrays in scenarios with large data volumes, avoiding performance bottlenecks caused by additional operations. At the same time, it ensures that subsequent processes can correctly parse empty arrays, avoiding exceptions caused by inconsistent data formats.

[0123] For example, to help understand the implementation process of the data migration method obtained by combining this embodiment with the above embodiment 1, please refer to Figure 6 , Figure 6 A brief flowchart of the data migration method is provided, specifically:

[0124] First, configure the data retrieval conditions and comparison conditions. The data retrieval conditions include, but are not limited to, the target table, field mapping rules, filter conditions, and step fields. The comparison conditions include at least the comparison fields.

[0125] Then, according to the data retrieval target table, a table in the target database is determined as the target table, which will receive the data to be added.

[0126] Determine the source table rows to be extracted based on preset filtering conditions, such as date ranges and specific field values. If the source table data volume is large, introduce stepping fields (such as timestamps) for statistics and paging. Incorporate the paging results of the stepping fields into the final data retrieval query statement to implement conditional filtering and paging, gradually extracting source table row data.

[0127] Next, based on business needs, the database identifies whether the fields of the data to be added have changed. If the fields in the source table row are unchanged, the source table row fields are mapped one-to-one with the fields in the target table row. If the fields in the source table row are changed, the fields in the source table row are converted to the format required by the target table according to pre-set field mapping rules. SQL functions are used to perform the necessary data conversion.

[0128] Subsequently, in order to implement incremental data synchronization, it is necessary to compare the data of the two databases row by row to obtain the newly added, unchanged, and deleted data, so as to avoid performing full database deletion and full database addition operations on the target database. Figure 7 , Figure 7The following is a flowchart for comparing data differences. In this embodiment, key field values for the source table row data and the target table row data are first extracted and compared. If the comparison result shows that the key field values for the extracted source table row data and the target table row data are the same, the data is marked as existing data and confirmed as data to be deleted. If the comparison result shows that the key field values for the extracted source table row data and the target table row data are different, the data is marked as data to be added.

[0129] Finally, generate and execute SQL insert statements to insert the data to be added into the target table and delete the data to be deleted.

[0130] Through the above-mentioned embodiment method, in order to achieve heterogeneous and incremental synchronization of data, the source table row data is retrieved and filtered according to the preset data retrieval conditions and filtering conditions, and the source table row data is mapped to the target format when retrieving data. The data differences between different libraries are compared through comparison fields, and high-performance data retrieval of the source library is achieved through stepping fields.

[0131] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data migration method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0132] This application also provides a data migration device, please refer to Figure 8 , the data migration device includes:

[0133] The data acquisition module 10 is used to obtain source table row data from the data source library according to the preset data acquisition conditions;

[0134] A comparison module 20 is used to compare the source table row data with the preset target table row data according to preset comparison conditions to obtain data to be added and data to be deleted;

[0135] The processing module 30 is configured to insert the data to be added into the target database and delete the data to be deleted.

[0136] The data migration device provided in this application, utilizing the data migration method described in the aforementioned embodiments, can address the technical problem of improving heterogeneous data migration efficiency while reducing migration costs. Compared to the prior art, the data migration device provided in this application achieves the same beneficial effects as the data migration method described in the aforementioned embodiments. Other technical features of the data migration device are the same as those disclosed in the aforementioned embodiments and are not further elaborated upon here.

[0137] The present application provides a data migration device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data migration method in the above-mentioned embodiment one.

[0138] Reference below Figure 9 , which shows a schematic structural diagram of a data migration device suitable for implementing embodiments of the present application. The data migration device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The data migration device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0139] like Figure 9 As shown, the data migration device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the data migration device. Processing device 1001, read-only memory 1002, and random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. Communication device 1009 can allow the data migration device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a data migration device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0140] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0141] The data migration device provided in this application, utilizing the data migration method described in the aforementioned embodiments, can address the technical problem of improving heterogeneous data migration efficiency while reducing migration costs. Compared to the prior art, the data migration device provided in this application achieves the same beneficial effects as the data migration method described in the aforementioned embodiments. Other technical features of the data migration device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0142] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0143] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0144] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the data migration method in the above-mentioned embodiment.

[0145] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0146] The computer-readable storage medium may be included in the data migration device, or may exist independently without being assembled into the data migration device.

[0147] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data migration device, the data migration device: obtains source table row data from a data source library according to preset data retrieval conditions; compares the source table row data with preset target table row data according to preset comparison conditions to obtain data to be added and data to be deleted; inserts the data to be added into the target database, and deletes the data to be deleted.

[0148] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0149] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0151] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned data migration method. This computer-readable storage medium addresses the technical problem of improving the efficiency of heterogeneous data migration while reducing migration costs. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the data migration method provided in the aforementioned embodiments, and are not further elaborated here.

[0152] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data migration method when executed by a processor.

[0153] The computer program product provided in this application can solve the technical problem of how to improve the efficiency of heterogeneous data migration while reducing migration costs. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the data migration method provided in the above embodiment, and will not be repeated here.

[0154] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data migration method, characterized in that: The data migration method includes: According to the preset data retrieval conditions, the source table row data is extracted from the data source library; Compare the source table row data with the preset target table row data according to the preset comparison conditions to obtain the data to be added and the data to be deleted; Insert the data to be added into the target database, and delete the data to be deleted.

2. The data migration method according to claim 1, wherein: The preset data retrieval conditions include at least a step field, a data retrieval filter condition, and a data retrieval field mapping. The step of retrieving data from a data source library to obtain source table row data according to the preset data retrieval conditions includes: According to the step field, the data source library is fetched page by page to obtain the initial source table row data; When an instruction to execute the data retrieval filtering condition is identified, the initial source table data is filtered according to the data retrieval filtering condition to obtain filtered source table row data; According to the fetch field mapping, the filtered source table row data is format mapped to obtain source table row data in a target format.

3. The data migration method according to claim 2, wherein: The step of fetching data page by page from the data source library according to the step field to obtain the initial source table row data includes: Counting the total amount of data in the data source library that meets the data retrieval filtering conditions to determine the paging size; Generate a paging query condition based on the value range of the step field and the paging size; According to the paging query conditions, data is extracted from the data source library in sequence to obtain initial source table data.

4. The data migration method according to claim 2, wherein: After the step of fetching data page by page from the data source library according to the step field to obtain initial source table row data, the method further includes: When no instruction for executing the data retrieval filtering condition is identified, a corresponding empty array is configured for the source table row data as a filtering result identifier.

5. The method according to claim 3, wherein The step of performing format mapping on the filtered source table row data according to the acquired field mapping to obtain the source table row data in a target format includes: Determine whether the fields of the filtered source table row data have changed based on business needs; If the fields of the filtered source table row data do not change, the fields of the filtered source table row data are matched one-to-one with the fields of the preset target table row data; If the fields of the filtered source table row data change, the keys and values in the fields of the filtered source table row data are mapped to the target format according to the preset field mapping rules and conditional judgment syntax to obtain the source table row data in the target format.

6. The data migration method according to claim 1, wherein: The step of comparing the source table row data with the preset target table row data according to the preset comparison conditions to obtain the data to be added and the data to be deleted includes: Extracting the key field values of the source table row data and the key field values of the preset target table row data; Determine whether the key field value of the source table row data is the same as the key field value of the preset target table row data; If the key field value of the source table row data is different from the key field value of the preset target table row data, marking the source table row data as the to-be-added data; If the key field value of the source table row data is the same as the key field value of the preset target table row data, the source table row data is marked as existing data, and the existing data is confirmed to be data to be deleted.

7. A data migration device, characterized in that: The data migration device includes: The data retrieval module is used to retrieve source table row data from the data source library according to the preset data retrieval conditions; A comparison module is used to compare the source table row data with the preset target table row data according to preset comparison conditions to obtain data to be added and data to be deleted; The processing module is used to insert the data to be added into the target database and delete the data to be deleted.

8. A data migration device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data migration method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

Images