Heterogeneous data online migration method, system and storage medium

Abstract

The present disclosure relates to the technical field of database system application, and provides a method, system and storage medium for online migration of heterogeneous data. The method includes: performing association calculation between the main table and all sub-tables for the read dynamic Table, and grouping and aggregating all the main tables and sub-tables according to the key of the main table; judging whether each Table is the main table, if so, selecting the fields of the main table to output to NoSQL (non-relational database), if not, processing the sub-table into a nested object array and outputting it to NoSQL (non-relational database). The present disclosure realizes the real-time online migration of a large amount of heterogeneous data, quickly, simply and effectively handles the update operations between dependent data, reduces the complexity caused by the dependency and arrival order between data change events, and realizes the effective output of heterogeneous data to NOSQL (non-relational database).

Description

Technical Field

[0001] The present disclosure relates to the technical field of database system application, and in particular to a method, system and storage medium for online migration of heterogeneous data. Background Art

[0002] In order to adapt to the needs of business development, the relational database used in the system needs to be replaced with NOSQL (non-relational database), and the work involved includes data migration and code transformation.

[0003] There are several traditional methods for data migration involving database replacement: 1. Shut down for maintenance, migrate all data, and switch to the new program. The disadvantage is that the system is unavailable during the migration, which is unacceptable for a system that is frequently used by C-end users. 2. Mainstream databases all support the CDC mechanism, which captures every row of data changes in the database and writes corresponding programs to process them. In application scenarios, various change events need to be processed and finally processed into a complex nested object.

[0004] The difficulties are: a) Data is dependent on each other (subtable depends on main table), while change events are independent. Dependent data needs to be loaded when processing. b) Change event types for each table include INIT, INSERT, UPDATE, and DELETE, which need to be processed separately. c) The order of events is unpredictable, and when processing, it is necessary to determine whether the dependent data has been processed. d) The same event may be consumed repeatedly, and event processing needs to support idempotence. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide a method, system and storage medium for online migration of heterogeneous data to solve the complexity of real-time online migration, update operations between dependent data, dependencies between data change events and arrival order, and output problems of NOSQL (non-relational database) in the prior art.

[0006] A first aspect of an embodiment of the present disclosure provides a method for online migration of heterogeneous data, characterized by comprising:

[0007] Submit a task to request online migration of heterogeneous data to NOSQL (non-relational database);

[0008] Parse heterogeneous data and read dynamic tables in heterogeneous data;

[0009] For the dynamic table read, the association calculation between the main table and all sub-tables is performed in the preset execution order, and each main table and sub-table is grouped and aggregated according to one or more keys of the main table;

[0010] According to the preset execution mode, the main table and the sub-table after the association calculation and grouping aggregation are processed into a nested object array and output to NoSQL (non-relational database), wherein the number of nesting is 0, 1, or multiple times; or it is determined whether each Table is the main table, if so, the fields of the main table are selected and output to NoSQL (non-relational database), if not, the sub-table is processed into a nested object array and output to NoSQL (non-relational database);

[0011] The online data migration task is completed.

[0012] A second aspect of an embodiment of the present disclosure provides a heterogeneous data online migration system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the related methods of the first aspect of the embodiment when executing the computer program.

[0013] According to a third aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0014] Compared with the prior art, the disclosed embodiments have the following advantages: real-time online migration of large amounts of heterogeneous data is achieved through associative calculation and group aggregation, update operations between dependent data are processed quickly, simply and effectively, and effective output to NOSQL (non-relational database) is achieved through complex nesting. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure;

[0017] Figure 2 It is a flowchart of a heterogeneous data online migration method provided by an embodiment of the present disclosure;

[0018] Figure 3 It is a flowchart of another heterogeneous data online migration method provided by an embodiment of the present disclosure;

[0019] Figure 4 It is a flowchart of another method for online migration of heterogeneous data provided by an embodiment of the present disclosure;

[0020] Figure 5 It is a flowchart of another method for online migration of heterogeneous data provided by an embodiment of the present disclosure;

[0021] Figure 6-1 It is a structural diagram of hash value calculation and balancing processing preparation of each computing unit provided by an embodiment of the present disclosure;

[0022] Figure 6-2 is a structural schematic diagram of a balancing processing process of each computing unit provided in an embodiment of the present disclosure;

[0023] Figure 7 is a structural diagram of a system provided by an embodiment of the present disclosure; DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0025] A method, system and storage medium for online migration of heterogeneous data according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.

[0027] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the electronic devices described above. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0028] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0029] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.

[0030] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0031] The user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, data, etc.

[0032] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.

[0033] Figure 2 It is a flowchart of a heterogeneous data online migration method provided by an embodiment of the present disclosure. Figure 2 The online migration method of heterogeneous data can be Figure 1 The terminal device or server executes. Figure 2 As shown, the heterogeneous data online migration method includes:

[0034] S201, submit NOSQL (non-relational database) task,

[0035] S202, parsing heterogeneous data,

[0036] S203, read the dynamic table,

[0037] S204, association calculation,

[0038] S205, group aggregation,

[0039] S206, processing each table into a nested object array,

[0040] S207, the online data migration task is completed.

[0041] The method steps are represented by "SXXX", where XXX is a three-digit consecutive number, such as S201, S202, S203, S204, S205, S206, and S207 of the method steps. It should be understood that the size of the serial number of each step in the embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the disclosed embodiment. The understanding of the method steps in other specific implementations is the same as that in this implementation, and will not be repeated.

[0042] Figure 3 It is a flowchart of another heterogeneous data online migration method provided by an embodiment of the present disclosure. Figure 3 The online migration method of heterogeneous data can be Figure 1 The terminal device or server executes. Figure 3 As shown, the heterogeneous data online migration method includes:

[0043] S301, submit NOSQL task,

[0044] S302, parsing heterogeneous data,

[0045] S303, read the dynamic table,

[0046] S304, group aggregation,

[0047] S305, association calculation,

[0048] S306, processing each table into a nested object array,

[0049] S307: The online data migration task is completed.

[0050] Figure 4 It is a flowchart of another method for online migration of heterogeneous data provided by an embodiment of the present disclosure. Figure 4 The online migration method of heterogeneous data can be Figure 1 The terminal device or server executes. Figure 4 As shown, the heterogeneous data online migration method includes:

[0051] S401, submit NOSQL task,

[0052] S402, parsing heterogeneous data,

[0053] S403, read the dynamic table,

[0054] S404, association calculation,

[0055] S405, group aggregation,

[0056] S406, determine whether the table after grouping and aggregation is the main table,

[0057] S407, if it is data from the main table, directly get the corresponding field of the main table,

[0058] S408: If it is sub-table data, multiple rows of data need to be concatenated into an array, that is, the sub-table is processed into a nested object array.

[0059] S409: Output the processed main table and sub-table data to NOSQL.

[0060] S410: The online data migration task is completed.

[0061] Figure 5 It is a flowchart of another method for online migration of heterogeneous data provided by an embodiment of the present disclosure. Figure 5 The online migration method of heterogeneous data can be Figure 1 The terminal device or server executes. Figure 5 As shown, the heterogeneous data online migration method includes:

[0062] S501, submit NOSQL task,

[0063] S502, parsing heterogeneous data,

[0064] S503, read the dynamic table,

[0065] S504, group aggregation,

[0066] S505, association calculation,

[0067] S506, determine whether the table after grouping and aggregation is the main table,

[0068] S507, if it is data from the main table, directly get the corresponding field of the main table,

[0069] S508: If it is sub-table data, multiple rows of data need to be concatenated into an array, that is, the sub-table is processed into a nested object array.

[0070] S409: Output the processed main table and sub-table data to NOSQL.

[0071] S410: The online data migration task is completed.

[0072] in, Figure 2-Figure 5In the process of heterogeneous data online migration method, submitting NOSQL task is to request online migration of heterogeneous data to NOSQL (non-relational database); in association calculation: the association calculation of the main table and all sub-tables is performed separately. In the grouping and aggregation process, each main table and sub-table is grouped and aggregated according to one or more keys of the main table.

[0073] According to one of the preset execution modes, the main table and the sub-table after the association calculation and grouping aggregation are processed into a nested object array and output to NoSQL (non-relational database) respectively, wherein the number of nesting is 0, 1, or multiple times; according to another preset execution mode, it is determined whether each Table is the main table. If so, the fields of the main table are selected and output to NoSQL (non-relational database); if not, the sub-table is processed into a nested object array and output to NoSQL (non-relational database).

[0074] Specifically, in order to replace heterogeneous data with NOSQL data, the calculation methods of association calculation and group aggregation are adopted. Association calculation can synchronously update the data in each table of heterogeneous data. By taking the main table as the benchmark, the common fields or related keys in each table are matched and connected to achieve data consistency processing; group aggregation can classify the data of each table based on the common main table fields or keys, and divide the metadata into multiple groups using specific conditions. Aggregation performs certain operations on the data in each group, and finally integrates the calculation results. The grouping and aggregation process is roughly divided into three steps:

[0075] Split: Split the data set into several groups according to certain criteria.

[0076] Apply: Apply a function or method to each group.

[0077] Merge: Integrate the resulting new value into the result object.

[0078] After the association calculation and grouping aggregation of each TABLE table, each TABLE table needs to be output to the NOSQL database and converted into the data format corresponding to the NOSQL database. Therefore, the processing method of converting heterogeneous data into NOSQL data is adopted: the corresponding heterogeneous data is processed into a nested object array. Since the main table is already NOSQL data, it is possible to increase the judgment of whether the TABLE table is the main table, and the main table is not processed and directly output to the NOSQL database. Alternatively, it is possible to not judge whether it is the main table and directly process the nested object array for all TABLE tables to simplify the program and directly output the nested object array to the NOSQL database.

[0079] According to the technical solution provided by the disclosed embodiments, by adopting the Table data processing method of associative calculation and grouping aggregation, high-speed data processing and synchronous associative update of a large amount of heterogeneous data can be achieved, real-time online migration of data can be realized, and update operations between dependent data can be quickly, simply and effectively processed. In addition, the output to NOSQL can be achieved by directly nesting the complex tables of each TABLE. It is also possible to determine whether it is the main table. If it is the main table, the main table fields are directly selected to output to NOSQL. If it is a sub-table, the sub-table is processed into a nested object array and output to NOSQL. A fast and effective data processing method for outputting to NOSQL is provided, and the method of not determining the main table can also simplify the program.

[0080] In some embodiments, the heterogeneous data online migration method uses a left outer join association calculation between the main table and all sub-tables when performing an association calculation between the main table and all sub-tables: taking the main table as a benchmark, the data of all sub-tables are matched and connected with the main table respectively to perform a left outer join association calculation between the main table and all sub-tables.

[0081] Specifically, the left outer join is based on the left table, connecting the data of the two tables, and then displaying NULL if there is no corresponding item in the left table. That is, according to the records of the left table, find the records that meet the conditions in the connected right table to match them. If no match is found with the left table, it is represented by null. Features of left join: Left outer join is a type of outer join, which will display all items in the left table, even if the data in some items is not fully filled (no corresponding value is found in the right table).

[0082] According to the technical solution provided by the embodiment of the present disclosure, data association and matching are achieved through left outer join, and data update processing is completed efficiently.

[0083] In some embodiments, the heterogeneous data online migration method processes the change event through a custom function during the process of reading the dynamic table, or after reading the dynamic table and before the associated calculation, or after reading the dynamic table and after the associated calculation, or after reading the dynamic table and before grouping and aggregation, or after reading the dynamic table and after grouping and aggregation, or after reading the dynamic table and before outputting to NoSQL, and further converts the read data or calculation results into the processing results of the change event. The heterogeneous data online migration method returns one or more object arrays when the function returns the result.

[0084] In the custom function, the heterogeneous data online migration method adds each row of data or all data to the result array after receiving the data, and removes duplicates according to one or more keys.

[0085] In the custom function of the heterogeneous data online migration method, when the data change event type is update, it is deleted from the result array according to one or more keys, and the adding operation is re-executed, or after the adding operation is executed, it is deleted from the result array according to one or more keys; when the data change event type is delete, it is necessary to delete from the result array according to one or more keys.

[0086] Specifically, the custom function defines the processing process of the change event, sets the function return result to be an array after the event processing is completed, ensures that the data is returned as an array after processing, and ensures the continuity of program processing; sets the deduplication operation after receiving a row of data or receiving all data each time, which can promptly and effectively handle data duplication problems; through the effective processing of change events, the processing of change events can be completed efficiently.

[0087] In some embodiments, for distributed computing involving multiple nodes, the heterogeneous data online migration method performs a merge operation on the object arrays transmitted from other nodes based on one or more keys to remove duplication. In order to ensure that the computing amount of each node is average, the heterogeneous data online migration method balances the computing task load to different computing units through hash values.

[0088] Figure 6-1 Schematic diagram of hash value calculation and balancing processing preparation of each computing unit provided in the embodiment of the present disclosure. Figure 6-1 As shown, each unit of this embodiment includes: main table 601, sub-table 1 (602), sub-table 2 (603), sub-table 3 (604), sub-table N (605), association calculation unit 1 (606) for performing association calculation between the main table and all sub-tables, unit N1 (607) for performing association calculation between the main table and all sub-tables, unit 1 (608) for grouping and aggregating each table according to the key of the main table, unit N2 (609) for grouping and aggregating each table according to the key of the main table, nested object array processing unit 1 (610) for each table / sub-table, and nested object array processing unit N3 (611) for each table / sub-table.

[0089] By calculating the hash values ​​of each TABLE table: the hash value of the main table 601: hash, the hash value of sub-table 1 (602): hash1, the hash value of sub-table 2 (603): hash2, the hash value of sub-table 3 (604): hash3, the hash value of sub-table N (605): hashN, the hash value of the association calculation unit 1 (606) between the main table and all sub-tables: association calculation hash1, the hash value of the association calculation unit N1 (607) between the main table and all sub-tables: association calculation hashN1, each table is grouped according to the key of the main table The hash value of aggregation unit 1 (608) is group aggregation hash1, each table implements the hash value of group aggregation unit N2 (609) according to the key of the main table: group aggregation hashN2, the hash value of the nested object array processing unit 1 (610) of each table / sub-table is nested processing hashN1, the hash value of the nested object array processing unit N3 (611) of each table / sub-table is nested processing hashN3, and the computing tasks are balanced to different computing units: according to all the hash values ​​mentioned, all hash values ​​are arranged on the hash value ring and balanced to each computing unit.

[0090] Figure 6-2 Schematic diagram of hash value balancing processing of each computing unit provided in the embodiment of the present disclosure. Figure 6-2 As shown, in Figure 6-1 On the basis of , the balancing process of each computing unit is added: balancing processing 1-1 of the association computing unit 1 (606) between the main table and all the sub-tables, balancing processing 1-N1 of the association computing unit N1 (607) between the main table and all the sub-tables, balancing processing 2-1 of the grouping aggregation unit 1 (608) of each table according to the key of the main table, balancing processing 2-N2 of the grouping aggregation unit N2 (609) of each table according to the key of the main table, balancing processing 3-1 of the nested object array processing unit 1 (610) of each table / sub-table, balancing processing 3-N3 of the nested object array processing unit N3 (611) of each table / sub-table; on the hash ring, for the three types of computing units (association computing, grouping aggregation, nested processing), the association computing units are balanced, the grouping aggregation units are balanced, and the nested processing units are balanced.

[0091] Specifically, the balancing process is as follows: On the hash ring, the balancing process of the associated calculation unit is as follows: According to the clockwise position relationship between the hash values ​​of the main table and the sub-table 1 on the hash ring, the hash value of the main table is in the clockwise direction of the hash value of the sub-table 1 on the hash ring. Therefore, proceed in the clockwise direction of the main table and select the nearest associated calculation unit 1 (606) for balancing. The balancing process in the figure is as follows: connecting lines are drawn from the position of the hash value (hash) of the main table and the position of the hash value (hash1) of the sub-table 1 respectively and connected to the balancing processing 1-1 of the associated calculation unit 1 (606) in the clockwise direction (the connecting lines are thin lines, and in the figure, the balancing processing lines of the associated calculation are all thin lines), completing the balancing processing of the associated calculation between the main table and the sub-table 1; according to the relationship between the main table and the sub-table 2's hash value is in the clockwise direction on the hash ring, and the hash value of sub-table 2 is in the clockwise direction of the hash value position of the main table on the hash ring. Therefore, advance in the clockwise direction of sub-table 2 and select the nearest associated calculation unit N1 (607) for balancing. The balancing process in the figure is as follows: connecting lines are drawn at the position of the hash value (hash) of the main table and the position of the hash value (hash2) of sub-table 2 respectively and connected with the balancing processing 1-N1 of the associated calculation unit sub-table N1 (607) in the clockwise direction to complete the balancing processing of the associated calculation between the main table and sub-table 2; the balancing processing of the associated calculation between the main table and sub-table 3 is the same as the balancing processing of the associated calculation between the main table and sub-table 2; the balancing processing of the associated calculation between the main table and sub-table N is the same as the balancing processing process of the associated calculation between the main table and sub-table 2, which will not be repeated here.

[0092] The balanced processing of grouping aggregation and nesting processing is based on the hash value of each table (main table and all sub-tables) in turn. After finding the corresponding position on the hash value ring, it advances in the clockwise direction and selects the nearest grouping aggregation unit to perform the balanced allocation operation of grouping aggregation (the connecting line is a medium thick line), and selects the nearest nesting processing unit to perform the balanced allocation operation of nesting processing (the connecting line is a large thick line). I will not go into details here.

[0093] The task load of the associated calculation is balanced to the corresponding N1 associated calculation units, the task load of the group aggregation is balanced to the corresponding N2 group aggregation units, and the task load of the nested object array processing calculation is balanced to different N3 nested object array processing units. The calculation task load of the calculation unit is balanced by hash value, which effectively realizes the optimization allocation of the overall calculation task.

[0094] In some embodiments, specifically, for a multi-node distributed system, a deduplication operation process is used to perform redundancy removal in a timely manner, thereby enabling remote data reception and ensuring efficient computing processing.

[0095] In some embodiments, the heterogeneous data online migration method synchronously sends the corresponding output nested object array to NoSQL (non-relational database) message to the communication channel during the process of executing the nested object array output to NoSQL (non-relational database), and the migration program subscribes to the communication channel information to process the synchronization logic. Preferably, the communication channel is a message queue.

[0096] Specifically, the data to be output to NOSQL is decoupled, peak-cut, and asynchronously processed through communication channels, especially message queues. Another function is to improve the receiving performance.

[0097] In some embodiments, the heterogeneous data online migration method captures the changed data of the heterogeneous data based on the CDC mechanism of the database during the process of reading the dynamic Table, generates data change events according to the captured results of the changed data, and collects data change events in real time.

[0098] Specifically, CDC (Change Data Capture) only needs to extract data that has changed within a period of time. The extraction and subsequent conversion and loading operations will obviously become more efficient because the amount of data that needs to be processed will be much smaller.

[0099] According to the technical solution provided by the embodiment of the present disclosure, data change events are monitored through the CDC mechanism to obtain changed data, and data acquisition is more efficient and faster.

[0100] In some embodiments, the heterogeneous data online migration method uses a streaming computing engine to parse SQL, and in the process of collecting and analyzing heterogeneous data, obtains data from the data buffer in a streaming manner and quickly converts it into corresponding computing tasks.

[0101] Specifically, currently commonly used streaming real-time computing engines are divided into two categories: row-oriented and micro-batch-oriented. The representative of row-oriented streaming real-time computing engines is Apache Storm (stream processing framework / distributed real-time computing system focusing on extremely low latency), which is typically characterized by low latency but low throughput. The representative of micro-batch-oriented streaming real-time computing engines is Spark Streaming (real-time stream processing framework), which is typically characterized by high latency but high throughput.

[0102] The mainstream streaming data line is divided into four stages: 1. Data collection: responsible for real-time data collection from different data sources, 2. Data buffering: to balance the inequality between data collection rate and data processing rate, 3. Real-time analysis: streaming data from the data buffer and quickly completing data processing, 4. Result storage: storing the calculation results in an external system.

[0103] According to the technical solution provided in the embodiment of the present disclosure, a higher throughput rate can be achieved by parsing SQL through a streaming computing engine.

[0104] In some embodiments, one or more keys mentioned in the heterogeneous data online migration method are primary keys. By extending the usage functions of the relevant TABLE table keys, the function expansion of data processing can be achieved, and more accurate data processing results can be achieved.

[0105] In the application process of the vehicle user database of autonomous driving, the following 1 main table + 7 sub-tables are aggregated into a complex nested object in the NOSQL database.

[0106] Member document structure:

[0107] Member id: 123

[0108] Nickname: Xiao Ming

[0109] Vehicle information array:

[0110] —License plate: Beijing X123, Brand: A

[0111] —License plate: Beijing Y456, Brand: B

[0112] Authentication information array:

[0113] —Document type: ID card, document number: ABC

[0114] —Document type: Passport, Document number: DEF

[0115] Contact information array:

[0116] —Contact method: mobile phone, contact number: 185****3256

[0117] —Contact method: landline, contact number: 010-892******

[0118] —Contact method: Email, contact number: 9558*****@qq.com

[0119] Address information array:

[0120] —Address type: Household registration address, Province and city: Beijing, District: Haidian, Street: Zizhuyuan, Detailed information (building number, unit, house number): ******

[0121] —Address type: Residential address, Province and city: Beijing, District: Haidian, Street: Huayuan Road, Detailed information (building number, unit, house number): ******

[0122] Family information array:

[0123] — Personnel: Zhang**, Relationship: Husband and wife

[0124] —Personnel: Wang**, Brand: Father and Son

[0125] According to the technical solution provided in the embodiment of the present disclosure, the amount of code is converted from more than 8,000 lines of Java code to the definition of 8 dynamic tables and 45 lines of SQL processing scripts, the efficiency is improved by more than 10 times, and the implementation is completely decoupled from specific business operations.

[0126] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0127] Figure 7 Schematic diagram of system 7 provided in the embodiment of the present disclosure. Figure 7 As shown, the system 7 of this embodiment includes: a processor 701, a memory 702, and a computer program 703 stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program 703, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 701 executes the computer program 703, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0128] Exemplarily, the computer program 703 may be divided into one or more modules / units, which are stored in the memory 702 and executed by the processor 701 to complete the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 703 in the system 7.

[0129] The system 7 may be an electronic device such as a desktop computer, a notebook, a PDA, or a cloud server. The system 7 may include but is not limited to a processor 701 and a memory 702. Those skilled in the art will appreciate that Figure 7 It is only an example of system 7 and does not constitute a limitation of system 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the system may also include input and output devices, network access devices, buses, etc.

[0130] The processor 701 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0131] The memory 702 may be an internal storage unit of the system 7, for example, a hard disk or memory of the system 7. The memory 702 may also be an external storage device of the system 7, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the system 7. Further, the memory 702 may also include both an internal storage unit of the system 7 and an external storage device. The memory 702 is used to store computer programs and other programs and data required by the electronic device. The memory 702 may also be used to temporarily store data that has been output or is to be output.

[0132] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0133] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0135] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / systems and methods can be implemented in other ways. For example, the device / system embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0139] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A method for online migration of heterogeneous data. It is characterized in that include: Submit a task to request online migration of heterogeneous data to a non-relational database; Parse heterogeneous data and read dynamic tables in heterogeneous data; For the dynamic table read, the association calculation between the main table and all the sub-tables is performed respectively according to the preset execution order, and each main table and sub-table is grouped and aggregated according to one or more keys of the main table; the association calculation synchronously updates the data in each table of the heterogeneous data, and the grouping and aggregation classifies the data of each table based on the common main table field or key; According to the preset execution mode, the main table and the sub-table after the association calculation and grouping aggregation are processed into a nested object array and output to the non-relational database respectively, wherein the number of nesting is 0, 1, or multiple times; or it is determined whether each table is the main table, if so, the field of the main table is selected and output to the non-relational database, if not, the sub-table is processed into a nested object array and output to the non-relational database; The online data migration request task is completed.

2. The method for online migration of heterogeneous data according to claim 1, It is characterized in that According to a predetermined execution order, during the process of reading a dynamic table, or after reading a dynamic table and before associative calculation, or after reading a dynamic table and after associative calculation, or after reading a dynamic table and before grouping and aggregation, or after reading a dynamic table and after grouping and aggregation, or after reading a dynamic table and before outputting to a non-relational database, a change event is processed through a custom function, and the read data or the result of the calculation is further converted into a processing result of the change event, and the function returns a result as one or more object arrays.

3. The method for online migration of heterogeneous data according to claim 2, It is characterized in that Each time a row of data is received or all the data is received, it is added to the result array and deduplicated based on one or more keys; When the data change event type is update, delete from the result array according to one or more keys and re-execute the add operation, or after executing the add operation, delete from the result array according to one or more keys; When the data change event type is delete, it is necessary to delete from the result array based on one or more keys.

4. The method for online migration of heterogeneous data according to any one of claims 1 to 3, It is characterized in that The association calculation between the main table and all the sub-tables is performed by left outer join association calculation between the main table and all the sub-tables: taking the main table as a reference, the data of all the sub-tables are matched and connected with the main table respectively.

5. The method for online migration of heterogeneous data according to any one of claim 4, It is characterized in that Calculate the hash values ​​of the main table and all sub-tables, and then balance the task load of associated calculations to the corresponding associated calculation units according to all hash values, balance the task load of group aggregation to the corresponding group aggregation units, and balance the task load of nested object array processing calculations to different nested object array processing units.

6. The method for online migration of heterogeneous data according to claim 5, It is characterized in that During the execution of output to a non-relational database, the corresponding message is synchronously sent to the message queue, and the migration program subscribes to the communication channel information to handle the synchronization logic itself.

7. The method for online migration of heterogeneous data according to claim 6, It is characterized in that A streaming computing engine is used to parse heterogeneous data. In the process of collecting and analyzing heterogeneous data, data is obtained from the data buffer in a streaming manner and quickly converted into corresponding computing tasks.

8. The method for online migration of heterogeneous data according to claim 7, It is characterized in that In the process of reading dynamic tables, based on the database's change data capture mechanism, the change data of heterogeneous data is captured, and data change events are generated according to the capture results of the change data, and data change events are collected in real time.

9. A heterogeneous data online migration system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images