Data verification method based on unitized architecture data migration and related device

Abstract

The invention discloses a data verification method based on unitized architecture data migration and a related device, relates to the technical field of data processing, and is applied to a resource manager to acquire registration information of a to-be-migrated system, and the registration information comprises migration configuration, comparison configuration and fragmentation configuration; and in response to a data comparison request after the migration of the to-be-migrated system is completed, allocating a comparison server to the data tables under different units based on the fragment configuration, and sending the migration configuration and the comparison configuration to the comparison server, so that the comparison server executes a data comparison task on the corresponding data tables based on the migration configuration and the comparison configuration. A general scheme is provided for the data comparison process after data migration under the unitized architecture, the comparison process is managed and scheduled in a unified mode by means of the characteristics of the unitized architecture, and the efficient and ordered comparison process under the unitized architecture is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data verification method and related devices based on unitized architecture data migration. Background Art

[0002] The traditional centralized architecture with mainframes and databases as the core can no longer meet the growing demand for large-scale transactions and data processing. With the continuous development of the information technology industry and the accelerated deepening of the core system construction in the banking industry, major banks are gradually replacing foreign software with domestic basic software. At the same time, in order to enhance the scalability and maintainability of the system, they often adopt the solution of multi-site active construction in the software deployment architecture.

[0003] Among them, unitization is a solution for multi-site active-active. After the data is migrated from the source database to the distributed database, the data is inserted into different units through data sharding. This requires a quick comparison of the data in the source database and the data in the target database. Especially in the context of the shutdown and data migration of key financial backend systems in the unitized architecture, how to shorten the time of data migration and data comparison, reduce the impact time on customers, and speed up the service recovery speed has become an unavoidable problem for each system during shutdown and migration. When solving the above problems, each system often develops a set of data migration systems and data comparison systems, which will increase the manpower cost of the system. The comparison system is often designed without reusability, and it takes a long time to compare massive financial data, affecting the customer experience. Summary of the invention

[0004] In view of the above problems, the present application provides a data verification method and related device based on unitized architecture data migration to solve the problem of rapid verification after unitized data migration. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a data verification method based on unitized architecture data migration, the data verification method based on unitized architecture data migration is applied to a resource manager, and the data verification method based on unitized architecture data migration includes:

[0006] Obtaining registration information of the system to be migrated, the registration information including migration configuration, comparison configuration and sharding configuration;

[0007] In response to the data comparison request after the migration of the system to be migrated is completed, a comparison server is allocated to the data tables under different units based on the sharding configuration, and the migration configuration and the comparison configuration are sent to the comparison server, so that the comparison server performs the data comparison task on the corresponding data table based on the migration configuration and the comparison configuration.

[0008] In a possible implementation, the shard configuration includes shard descriptions and unit mapping descriptions of different data tables;

[0009] The data table allocation comparison server based on the shard configuration for different units includes:

[0010] For each unit, determine the first data table and its data volume under the unit according to the unit mapping description and the shard description;

[0011] Sorting the first data table in descending order of data amount;

[0012] Obtaining the server configuration of the unit, and determining the server cluster under the unit according to the server configuration;

[0013] The first data table is traversed according to the sorting result of the first data table, and a server with the smallest amount of data in the server cluster is selected as a comparison server of the traversed first data table until the first data table is completely traversed.

[0014] In a possible implementation, the comparison configuration includes a timestamp configuration;

[0015] Correspondingly, the comparison server performs a data comparison task on the corresponding data table based on the migration configuration and the comparison configuration, including:

[0016] The comparison server obtains a comparison history record table, wherein the comparison history record table contains comparison records of the corresponding data table in the executed data comparison task, and the comparison record contains the last comparison data item timestamp and the last comparison data item hash value; determines the candidate comparison data items of the corresponding data table in the data comparison task to be executed based on the timestamp configuration and the last comparison data item timestamp; determines the initial comparison data items of the corresponding data table from the candidate comparison data items according to the last comparison data item hash value, executes the data comparison task with the initial comparison data items, and updates the comparison history record table.

[0017] In a possible implementation, the data verification method based on unitized architecture data migration further includes:

[0018] In the process that the comparison server executes the data comparison task on the corresponding data table, the task forwarding of the distributed coordination service is performed in the comparison server.

[0019] In a possible implementation, the performing task forwarding of the distributed coordination service in the comparison server includes:

[0020] By first creating a temporary node in the permanent node to monitor the running status of the comparison server, the comparison server corresponds to the temporary node one by one;

[0021] Creating a task completion queue to record the servers in the comparison server that have completed the data comparison task;

[0022] If it is monitored that the running state of the first comparison server is offline and the first comparison server is not in the task completion queue, query the comparison history record table for a target comparison record of the second data table assigned to the first comparison server;

[0023] Determine, according to the target comparison record, a target comparison data item in the second data table for which the first comparison server has not performed a data comparison task, and allocate a second comparison server to the target comparison data item.

[0024] In a possible implementation, the data verification method based on unitized architecture data migration further includes:

[0025] Deleting temporary nodes corresponding to servers on which data comparison tasks have been completed based on the task completion queue;

[0026] When all temporary nodes corresponding to the comparison server are deleted, the permanent node is deleted.

[0027] A second aspect of the present application provides a data verification device based on unitized architecture data migration, the data verification device based on unitized architecture data migration is applied to a resource manager, and the data verification device based on unitized architecture data migration includes:

[0028] An information acquisition module, used to acquire registration information of the system to be migrated, wherein the registration information includes migration configuration, comparison configuration and sharding configuration;

[0029] A request response module is used to respond to the data comparison request after the migration of the system to be migrated is completed, allocate comparison servers to data tables under different units based on the sharding configuration, and send the migration configuration and the comparison configuration to the comparison server, so that the comparison server performs data comparison tasks on the corresponding data tables based on the migration configuration and the comparison configuration.

[0030] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the data verification method based on unitized architecture data migration according to the above-mentioned first aspect or any implementation method of the first aspect.

[0031] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program so that the electronic device can implement the data verification method based on unitized architecture data migration according to the above-mentioned first aspect or any implementation method of the first aspect.

[0034] The fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the data verification method based on unitized architecture data migration according to the above-mentioned first aspect or any implementation method of the first aspect.

[0035] By means of the above technical scheme, the data verification method and related devices based on unitized architecture data migration provided by the present application are applied to the resource manager to obtain the registration information of the system to be migrated, and the registration information includes migration configuration, comparison configuration and sharding configuration; in response to the data comparison request after the system to be migrated completes the migration, the comparison server is allocated to the data tables under different units based on the sharding configuration, and the migration configuration and the comparison configuration are sent to the comparison server, so that the comparison server performs the data comparison task on the corresponding data table based on the migration configuration and the comparison configuration. The present application provides a general solution for the data comparison process after data migration under the unitized architecture. By utilizing the characteristics of the unitized architecture, it is possible to realize the unified registration of migration information, centralized management of comparison server resources, issuance of comparison requests, comparison according to unit data, and unified integration and return of results, and to unify the management and scheduling of the comparison process, so as to realize the efficient and orderly comparison process under the unitized architecture, so that the system to be migrated only needs to care about the migration of business logic, improve the migration efficiency, and help the enterprise basic software upgrade. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0037] Figure 1 A flowchart of a data verification method based on unitized architecture data migration provided in an embodiment of the present application;

[0038] Figure 2 A partial flow chart of a data verification method based on unitized architecture data migration provided in an embodiment of the present application;

[0039] Figure 3 An example diagram of data slicing provided in an embodiment of the present application;

[0040] Figure 4 A partial flow chart of a data verification method based on unitized architecture data migration provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of the structure of a data verification device based on unitized architecture data migration provided by an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0044] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0046] To facilitate understanding of this application, the following first describes the relevant concepts involved in this application:

[0047] Remote and multi-active: Remote means building data centers in different geographical locations, and multi-active means that systems in different geographical locations can provide business services.

[0048] Unit: refers to a self-contained collection that can complete business operations. This collection includes all services and part of the data assigned to this unit.

[0049] Data Sharding: The full amount of data is divided horizontally according to a certain dimension. The data on each shard does not overlap. Generally, there are two sharding methods: horizontal sharding and vertical sharding. Horizontal sharding refers to storing different rows of the same table in different database nodes, and vertical sharding refers to storing different columns of the same table in different database nodes. This application mainly focuses on horizontal sharding.

[0050] Distributed coordination service: responsible for coordinating and managing the various components of the distributed system to ensure that the various components can work together efficiently and orderly.

[0051] When the financial backend system is transformed from a centralized architecture to a distributed unitized architecture, data migration and verification are required. With the explosive growth of financial technology, the data verification stage takes a long time. This application provides a data verification method based on unitized architecture data migration, which utilizes the unitized deployment characteristics to reduce cross-unit access and designs a general method to solve the problem of rapid verification after unitized data migration.

[0052] See also Figure 1 , Figure 1 A flowchart of a data verification method based on unitized architecture data migration provided by an embodiment of the present application. Figure 1 As shown, a data verification method based on unitized architecture data migration provided by an embodiment of the present application is applied to a resource manager and can be applicable to data verification of different databases, including steps S101 to S102, which are described in detail below. It should be noted that the relationship between the resource manager and the unit can be one-to-one or one-to-many, and the embodiment of the present application does not limit this. The resource manager can implement the data verification method of the present application on the unit to which it corresponds.

[0053] S101, obtaining registration information of the system to be migrated, where the registration information includes migration configuration, comparison configuration and sharding configuration.

[0054] In the embodiment of the present application, the database information before and after the migration of the system to be migrated in the enterprise can be entered through the registration of the system information to be migrated, and the registration information of the system to be migrated is obtained, and the registration information includes the migration configuration, the comparison configuration and the sharding configuration. Among them, the migration configuration includes the source database type and port address, the target database type and port address, the mapping of the source database and the target database table name, the table name mapping relationship (for the case of split tables), the primary key information, the table field mapping relationship, and the table migration batch. The comparison configuration includes the target table field to be compared, and the sharding configuration includes the sharding information (such as which field is used for sharding, and the mapping relationship between sharding and units).

[0055] S102, in response to the data comparison request after the migration of the system to be migrated is completed, a comparison server is allocated to the data tables under different units based on the sharding configuration, and the migration configuration and the comparison configuration are sent to the comparison server, so that the comparison server performs the data comparison task on the corresponding data table based on the migration configuration and the comparison configuration.

[0056] In the embodiment of the present application, the systems to be migrated complete the data migration of the database respectively, and after the migration is completed, a data comparison request is initiated. In response to this, a comparison server is allocated to the data table under each unit based on the shard configuration, and the migration configuration and the comparison configuration are sent to the comparison server. The data of a data table can be deployed in at least one unit, and the data of the data table in a unit can be deployed in at least one shard. In this regard, respective comparison servers can be allocated to different data tables according to the unit. Based on this, the comparison server can complete the data execution task of the corresponding data table based on the migration configuration and the comparison configuration.

[0057] See also Figure 2 , Figure 2 A partial flow chart of a data verification method based on unitized architecture data migration provided by an embodiment of the present application. Figure 2 As shown, an embodiment of the present application provides a data verification method based on unitized architecture data migration, wherein the shard configuration includes shard descriptions of different data tables (such as the fields of the data table in the shard, the size of the shard, etc.) and unit mapping descriptions (i.e., the mapping relationship between the shard and the unit), and step S102 "comparison servers are allocated to data tables under different units based on the shard configuration" may include steps S201 to S204, and these steps are described in detail below.

[0058] S201, for each unit, determine the first data table and its data volume under the unit according to the unit mapping description and the shard description.

[0059] S202, sorting the first data table in descending order of data volume.

[0060] S203, obtaining the server configuration of the unit, and determining the server cluster under the unit according to the server configuration.

[0061] S204 , traversing the first data table according to the sorting result of the first data table, selecting a server with the smallest amount of data in the server cluster as a comparison server for the traversed first data table, until the first data table is completely traversed.

[0062] See also Figure 3 , Figure 3 This is an example diagram of data slicing provided in an embodiment of the present application. Figure 3As shown, the data of a table in the database before migration is distributed on different shards after migration, each shard contains a subset of the total data set, and these shards are distributed in different units. Take Table 1 as an example, which contains four rows of records in the database before migration. After migration, Table 1 has two rows of records in unit 1 and two rows of records in unit 2. Based on this, this application performs unit-by-unit calculation allocation comparison on the server based on the characteristics of data shard 1.

[0063] Specifically, in an embodiment of the present application, for each unit, the data table (i.e., the first data table) under the unit and its data volume can be determined according to its unit mapping description and shard description, wherein the data volume = the number of rows of data in the data table in the unit × the number of fields, and the first data table is sorted in descending order of data volume, so as to obtain the data volume queue corresponding to the unit.

[0064] For ease of understanding, it is assumed that the data tables under the unit are determined to include data table 1 to data table 6 according to the unit mapping description and the shard description, and the data volume of each of data table 1 to data table 6 is determined. Data table 1 to data table 6 are sorted in descending order of data volume to obtain the data volume queue data_queue[{data table 1: 10000},{data table 2: 8000},{data table 3: 6000},{data table 4: 5500},{data table 5: 4000},{data table 6: 1000}]. Taking {data table 1: 10000} as an example, it means that the data volume of data table 1 is 10000.

[0065] Further, the server configuration of the unit is obtained, and the server cluster under the unit is determined according to the server configuration. The server cluster is composed of multiple servers. The comparison allocation queue is initialized based on the service period cluster. The size of the comparison allocation queue is the same as the number of servers in the service period cluster. For example, assuming that the server cluster is composed of server 1 to server 3, the comparison allocation queue allocation=[{server 1: 0},{server 2: 0},{server 3: 0}], taking {server 1: 0} as an example, it means that the amount of data of server 1 in the initial state is 0. In addition, the comparison allocation queue is empty and its comparison task queue tasks=[[],[],[]], which means that the shards allocated to servers 1 to server 3 are empty in the initial state.

[0066] Traverse the data queue data_queue, pop up the current queue head {data table 1: 10000}, and assign it to the server with the smallest data volume (assuming it is server 1). At this time, compare the allocation queue allocation=[{server 1: 0+10000},{server 2: 0},{server 3: 0}, and compare the task queue tasks=[[data table 1],[],[]]. Remove {data table 1: 10000} from the data queue data_queue. At this time, the data queue data_queue is updated to [{data table 2: 8000},{data table 3: 6000},{data table 4: 5500},{data table 5: 4000},{data table 6: 1000}]

[0067] Continue to traverse the data queue data_queue, pop up the current queue head {data table 2: 8000}, and assign it to the server with the smallest data volume (assuming it is server 2). At this time, compare the allocation queue allocation=[{server 1: 0+10000},{server 2: 0+8000},{server 3: 0}, and compare the task queue tasks=[[data table 1],[data table 2],[]]. Remove {data table 2: 8000} from the data queue data_queue. At this time, the data queue data_queue is updated to [{data table 3: 6000},{data table 4: 5500},{data table 5: 4000},{data table 6: 1000}].

[0068] Repeat the above steps to allocate server 3 to data table 3, server 3 to data table 4, server 2 to data table 5, and server 1 to data table 6. Finally, the comparison allocation queue allocation=[{Server 1: 11000},{Server 2: 115000},{Server 3: 12000}, and the comparison task queue tasks=[[Data Table 1, Data Table 6],[Data Table 2, Data Table 5],[Data Table 3, Data Table 4]]. In this way, the allocation calculation of the comparison servers of each unit can be completed. The resource manager records the allocated tasks in the dispatched task table and records the task batches of the system to be migrated.

[0069] Further, the resource manager sends the machine number and comparison task of the comparison server to the comparison server assigned to each unit to start the comparison calculation. Based on the above comparison server allocation calculation results, the comparison task queue, migration configuration and comparison configuration of each comparison server are sent to the comparison server assigned to each data table.

[0070] The comparison server first obtains the source database driver according to the source database type and connects to the source database according to the source database port address, and then obtains the target database driver according to the target database type and connects to the target database according to the target database port address. Then, the data items of the data table assigned in the comparison allocation queue are processed in turn, and concurrent processing is possible here. When performing the comparison, a certain amount of data in the target table in the current unit is first read, and the data of the source database is queried according to the table name mapping relationship and the table field mapping relationship, and then the comparison is performed. At the same time, the currently compared quantity and inconsistent data are recorded in the memory.

[0071] In a possible implementation, considering that most financial transaction tables contain transaction time fields, and full data comparison requires more time, the embodiment of the present application provides an incremental comparison function on the basis of providing a full comparison function. Whether to adopt a full comparison or an incremental comparison is determined by the input parameters of the system to be migrated and the timestamp fields in the source and target tables configured by the system to be migrated. In this regard, the comparison configuration includes a timestamp configuration (including the timestamp fields and formats in the source and target tables).

[0072] In this regard, in the above step S30, "the comparison server performs a data comparison task on the corresponding data table based on the migration configuration and the comparison configuration", the following steps may be adopted:

[0073] The comparison server obtains a comparison history table, which contains comparison records of the corresponding data table in the executed data comparison tasks, and the comparison records contain the timestamp of the last comparison data item and the hash value of the last comparison data item; determines the candidate comparison data items of the corresponding data table in the data comparison task to be executed based on the timestamp configuration and the timestamp of the last comparison data item; determines the initial comparison data items of the corresponding data table from the candidate comparison data items according to the hash value of the last comparison data item, executes the data comparison task with the initial comparison data items, and updates the comparison history table.

[0074] In the embodiment of the present application, the so-called incremental comparison means that this comparison starts directly from where the previous comparison ended, instead of starting from the beginning. Considering that the system to be migrated may require multiple batches of migration processes, it will be extremely time-consuming to compare all the existing data after each migration. Most tables in financial data contain transaction time fields. For this purpose, the present application designs an incremental comparison method.

[0075] Each time the comparison server completes processing a data table in its task queue, it writes a comparison record to the history record table. The comparison record contains the unit number, machine number, task batch number, last comparison data item timestamp, last comparison data item hash value and table drop time.

[0076] First, determine whether the source table meets the incremental comparison conditions based on the source table timestamp field sent by the system to be migrated. When the source table timestamp field is empty, it is determined that the table does not meet the incremental comparison conditions and a full comparison is used directly. When the source table timestamp field is not empty, it is determined that the table meets the incremental comparison conditions and whether to perform an incremental comparison is determined based on the request parameters of the system to be migrated.

[0077] Secondly, when the parameters passed in by the system to be migrated indicate that an incremental comparison is required, after a comparison server obtains its own comparison task queue, it can query the comparison record most recently generated by the corresponding data table in this unit from the comparison history record table, take the timestamp of the last comparison data item in the comparison record as the starting point, perform field mapping on the timestamp field in the data table, query the comparison data items whose timestamp fields are later than the timestamp of the last comparison data item as candidate comparison data items for the data comparison task to be executed, perform hash operations on the candidate comparison data items, and match the hash operations results of the candidate comparison data items with the hash values ​​of the last comparison data items, use the matched comparison data items as the initial comparison data items of the data table, and then perform incremental queries using the timestamp field to execute the data comparison task.

[0078] Finally, when a comparison server finishes processing the last comparison data item of the corresponding shard, it generates a record to the comparison history record table. If there are inconsistent table records, the first inconsistent data item is recorded, otherwise the last data item is recorded to facilitate subsequent incremental comparison.

[0079] It should be noted that full comparison processing is to compare all data, which occurs in the following two scenarios: the source table of the system to be migrated has no timestamp field or the input parameters of the system to be migrated require full comparison. For scenarios with timestamp requirements for full comparison, full comparison, like incremental comparison, will record the data in the comparison history table at the end of the single table comparison to facilitate subsequent incremental comparison.

[0080] In addition, after the comparison server completes the comparison task, it returns the comparison results to the resource manager, which returns the results to the system to be migrated after unified integration. The system to be migrated can determine which data items are inconsistent based on the returned results.

[0081] In a possible implementation, in order to prevent a machine failure on a comparison server during the comparison process, resulting in the failure of comparison of some slice data, the embodiment of the present application designs a task forwarding strategy for a distributed coordination service. In this regard, the embodiment of the present application provides a data verification method based on unitized architecture data migration, which also includes the following steps:

[0082] In the process of the comparison server executing the data comparison task on the corresponding data table, the task forwarding of the distributed coordination service is performed in the comparison server.

[0083] See also Figure 4 , Figure 4 A partial flow chart of a data verification method based on unitized architecture data migration provided by an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides a data verification method based on unitized architecture data migration, wherein the above step of "forwarding tasks of distributed coordination services in comparison servers" may include steps S401 to S404, and these steps are described in detail below.

[0084] S401, a temporary node is first created in a permanent node to monitor the running status of a comparison server, and the comparison server corresponds to the temporary node one by one.

[0085] S402, creating a task completion queue to record the servers in the comparison server that have completed the data comparison task.

[0086] S403: If it is monitored that the running status of the first comparison server is offline and the first comparison server is not in the task completion queue, query the comparison history record table for a target comparison record of the second data table assigned to the first comparison server.

[0087] S404: Determine, according to the target comparison record, a target comparison data item in the second data table for which the first comparison server has not executed the data comparison task, and allocate a second comparison server to the target comparison data item.

[0088] In an embodiment of the present application, the resource manager completes the creation and monitoring of permanent nodes while issuing the comparison task. After the comparison task is issued to the comparison server, each comparison server creates a temporary node under the permanent node directory, and then starts to execute the data comparison task. It should be noted that the life cycle of the temporary node depends on the session that created them. Once the session ends, the temporary node will be automatically deleted; the life cycle of the permanent node does not depend on the session. After the comparison program on the comparison server is successfully started, a temporary node named machine number will be created in the directory of permanent node / cmp / migration system name / task batch number / unit number / . Any data is stored in the node. The permanent node is established by the resource manager, and the resource manager is responsible for monitoring all nodes under this directory. At the same time, the resource manager holds a task completion queue, which is used to record the machine number of the comparison server that has completed the data comparison task.

[0089] When a comparison server goes down, the resource manager can obtain the machine number of the comparison server after monitoring its offline according to the change of the temporary node to determine whether the comparison server is in the task completion queue. If not, the task (i.e., the second data table) sent to the comparison server is determined from the comparison task queue, and then the comparison record (i.e., the target comparison record) corresponding to the second data table is queried from the comparison history record table to determine the comparison data item (i.e., the target comparison data item) of the second data table that has not been executed for the data comparison task, and forward it as a task randomly or forward it to a comparison server with the least remaining comparison amount (i.e., the second comparison server). After obtaining the task, the second comparison server re-merges the target comparison record with its originally assigned data table and continues the comparison until the comparison process is completed.

[0090] On this basis, after completing the data comparison task, the comparison server can return the result to the resource manager, and the resource manager will record the machine number of the server in the task completion queue and delete its corresponding temporary node. If all temporary nodes are deleted, it means that all comparison servers under the unit have completed execution. At this time, the permanent node is deleted, including the permanent node directory.

[0091] The embodiment of the present application provides a data verification method based on unitized architecture data migration, and provides a general solution for the data comparison process after data migration under the unitized architecture. By utilizing the characteristics of the unitized architecture, it is possible to realize unified registration of migration information, centralized management of comparison server resources, issuing of comparison requests, comparison according to unit data, and unified integration and return of results, and unified management and scheduling of the comparison process to achieve efficient and orderly comparison process under the unitized architecture, so that the system to be migrated only needs to care about the migration of business logic, improve migration efficiency, and help upgrade the basic software of the enterprise. The present application has the following advantages:

[0092] 1) Targeted design is carried out according to the characteristics of data sharding, which greatly reduces the time consumption of data transmission network.

[0093] 2) Design an algorithm for comparison resource allocation so that the comparison tasks are distributed as evenly as possible and the comparison time overhead is reduced.

[0094] 3) An incremental data comparison method for financial transaction data is designed, which effectively reduces the data comparison time in multiple migration scenarios.

[0095] 4) It provides a universal method for comparing data after migration of different databases, avoiding repeated development of comparison tools for the system to be migrated and having good reusability.

[0096] 5) A method of task recalculation, server selection, and forwarding based on distributed coordination services is designed. When some machines fail during the comparison process, task forwarding can be achieved, ensuring the effectiveness and availability of the comparison process.

[0097] A data verification method based on unitized architecture data migration provided by an embodiment of the present application is introduced above. A device for executing the above-mentioned data verification method based on unitized architecture data migration will be introduced below.

[0098] See also Figure 5 , Figure 5 A schematic diagram of the structure of a data verification device based on unitized architecture data migration provided by an embodiment of the present application. Figure 5 As shown, a data verification device based on unitized architecture data migration provided by an embodiment of the present application is applied to a resource manager, including:

[0099] An information acquisition module 10 is used to acquire registration information of the system to be migrated, the registration information including migration configuration, comparison configuration and sharding configuration;

[0100] The request response module 20 is used to respond to the data comparison request after the migration of the system to be migrated is completed, allocate comparison servers to the data tables under different units based on the sharding configuration, and send the migration configuration and the comparison configuration to the comparison server, so that the comparison server performs the data comparison task on the corresponding data table based on the migration configuration and the comparison configuration.

[0101] In a possible implementation, the shard configuration includes shard descriptions and unit mapping descriptions of different data tables;

[0102] The request response module 20 for allocating and comparing servers for data tables under different units based on the shard configuration is specifically used for:

[0103] For each unit, the first data table under the unit and its data volume are determined according to the unit mapping description and the sharding description; the first data table is sorted in descending order of data volume; the server configuration of the unit is obtained, and the server cluster under the unit is determined according to the server configuration; the first data table is traversed according to the sorting result of the first data table, and the server with the smallest data volume in the server cluster is selected as the comparison server of the traversed first data table, until the first data table is traversed.

[0104] In a possible implementation, the comparison configuration includes a timestamp configuration;

[0105] Accordingly, the comparison server performs data comparison tasks on the corresponding data tables based on the migration configuration and the comparison configuration, including:

[0106] The comparison server obtains a comparison history table, which contains comparison records of the corresponding data table in the executed data comparison tasks, and the comparison records contain the timestamp of the last comparison data item and the hash value of the last comparison data item; determines the candidate comparison data items of the corresponding data table in the data comparison task to be executed based on the timestamp configuration and the timestamp of the last comparison data item; determines the initial comparison data items of the corresponding data table from the candidate comparison data items according to the hash value of the last comparison data item, executes the data comparison task with the initial comparison data items, and updates the comparison history table.

[0107] In a possible implementation, the request response module 20 is further configured to:

[0108] In the process of the comparison server executing the data comparison task on the corresponding data table, the task forwarding of the distributed coordination service is performed in the comparison server.

[0109] In a possible implementation, the request response module 20 for performing task forwarding of a distributed coordination service in a comparison server is specifically used to:

[0110] By first creating a temporary node in the permanent node to monitor the running status of the comparison server, the comparison server and the temporary node correspond one to one; by creating a task completion queue to record the servers in the comparison server that have completed the data comparison task; if it is monitored that the running status of the first comparison server is offline and the first comparison server is not in the task completion queue, query the target comparison record of the second data table assigned to the first comparison server in the comparison history record; determine the target comparison data item in the second data table that has not been executed by the first comparison server for the data comparison task according to the target comparison record, and assign the second comparison server to the target comparison data item.

[0111] In a possible implementation, the request response module 20 is further configured to:

[0112] Based on the task completion queue, the temporary nodes corresponding to the servers on which the data comparison task has been completed are deleted; when all the temporary nodes corresponding to the comparison servers are deleted, the permanent nodes are deleted.

[0113] It should be noted that the detailed functions of each module in the embodiment of the present application can be found in the corresponding public part of the above-mentioned data verification method embodiment based on unitized architecture data migration, which will not be repeated here.

[0114] The present application also provides an electronic device. Figure 6 , Figure 6A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0117] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any data verification method based on unitized architecture data migration provided in the embodiment of the present application.

[0118] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any data verification method based on unitized architecture data migration provided in an embodiment of the present application.

[0119] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0120] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0121] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0122] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A data verification method based on unitized architecture data migration, characterized in that: The data verification method based on unitized architecture data migration is applied to a resource manager, and the data verification method based on unitized architecture data migration includes: Obtaining registration information of the system to be migrated, the registration information including migration configuration, comparison configuration and sharding configuration; In response to the data comparison request after the migration of the system to be migrated is completed, a comparison server is allocated to the data tables under different units based on the sharding configuration, and the migration configuration and the comparison configuration are sent to the comparison server, so that the comparison server performs the data comparison task on the corresponding data table based on the migration configuration and the comparison configuration.

2. The data verification method based on unitized architecture data migration according to claim 1, characterized in that: The sharding configuration includes sharding descriptions and unit mapping descriptions of different data tables; The data table allocation comparison server based on the shard configuration for different units includes: For each unit, determine the first data table and its data volume under the unit according to the unit mapping description and the shard description; Sorting the first data table in descending order of data amount; Obtaining the server configuration of the unit, and determining the server cluster under the unit according to the server configuration; The first data table is traversed according to the sorting result of the first data table, and a server with the smallest amount of data in the server cluster is selected as a comparison server of the traversed first data table until the first data table is completely traversed.

3. The data verification method based on unitized architecture data migration according to claim 1, characterized in that: The comparison configuration includes a timestamp configuration; Correspondingly, the comparison server performs a data comparison task on the corresponding data table based on the migration configuration and the comparison configuration, including: The comparison server obtains a comparison history record table, wherein the comparison history record table contains comparison records of the corresponding data table in the executed data comparison task, and the comparison record contains the last comparison data item timestamp and the last comparison data item hash value; determines the candidate comparison data items of the corresponding data table in the data comparison task to be executed based on the timestamp configuration and the last comparison data item timestamp; determines the initial comparison data items of the corresponding data table from the candidate comparison data items according to the last comparison data item hash value, executes the data comparison task with the initial comparison data items, and updates the comparison history record table.

4. The data verification method based on unitized architecture data migration according to claim 3 is characterized in that: The data verification method based on unitized architecture data migration also includes: In the process that the comparison server executes the data comparison task on the corresponding data table, the task forwarding of the distributed coordination service is performed in the comparison server.

5. The data verification method based on unitized architecture data migration according to claim 4, characterized in that: The task forwarding of the distributed coordination service in the comparison server includes: By first creating a temporary node in the permanent node to monitor the running status of the comparison server, the comparison server corresponds to the temporary node one by one; Creating a task completion queue to record the servers in the comparison server that have completed the data comparison task; If it is monitored that the running state of the first comparison server is offline and the first comparison server is not in the task completion queue, query the comparison history record table for a target comparison record of the second data table assigned to the first comparison server; Determine, according to the target comparison record, a target comparison data item in the second data table for which the first comparison server has not performed a data comparison task, and allocate a second comparison server to the target comparison data item.

6. The data verification method based on unitized architecture data migration according to claim 4, characterized in that: The data verification method based on unitized architecture data migration also includes: Deleting temporary nodes corresponding to servers on which data comparison tasks have been completed based on the task completion queue; When all temporary nodes corresponding to the comparison server are deleted, the permanent node is deleted.

7. A data verification device based on unitized architecture data migration, characterized in that: The data verification device based on unitized architecture data migration is applied to a resource manager, and the data verification device based on unitized architecture data migration includes: An information acquisition module, used to acquire registration information of the system to be migrated, wherein the registration information includes migration configuration, comparison configuration and sharding configuration; A request response module is used to respond to the data comparison request after the migration of the system to be migrated is completed, allocate comparison servers to data tables under different units based on the sharding configuration, and send the migration configuration and the comparison configuration to the comparison server, so that the comparison server performs data comparison tasks on the corresponding data tables based on the migration configuration and the comparison configuration.

8. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the data verification method based on unitized architecture data migration as described in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the data verification method based on unitized architecture data migration as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the data verification method based on unitized architecture data migration as described in any one of claims 1 to 6.

Images