A data checking method, system, device and storage medium

Abstract

The application discloses a data checking method, system, device and storage medium. The method comprises the following steps: a task transfer server determines a to-be-processed checking task corresponding to each data checking server in response to a checking task acquisition request sent by each data checking server, wherein the to-be-processed checking task comprises checking range information and data block parameters, and the data block parameters are determined by a task control server based on database delay information; each data checking server determines target to-be-checked data based on the checking range information in the corresponding to-be-processed checking task; data is blocked based on the data block parameters, and a plurality of data checking threads corresponding to the number of a plurality of to-be-checked data blocks are created, and data checking is performed on the plurality of to-be-checked data blocks respectively. The technical scheme provided by the application can combine the task transfer server and the plurality of data checking servers, greatly improve the efficiency and flexibility of data checking, and realize flexible and reliable concurrent control.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data verification method, system, device, and storage medium. Background Technology

[0002] As business data volumes grow, online business data is typically initially recorded in a database used for real-time business data management (such as a MySQL database). However, due to server capacity limitations, historical data is cleaned up periodically to free up available space. Therefore, after recording online business data in the database for real-time business data management, it can be synchronized to other types of databases (such as Elasticsearch or HBase) for subsequent business data querying and reuse. However, during data synchronization, issues such as missed or incorrect transmission may occur. Therefore, it is necessary to verify the data before and after transmission to ensure data reliability.

[0003] Current technologies for cross-database data reconciliation typically utilize a single, fixed server configured for the reconciliation process, pulling data from the database for comparison. This approach is inefficient and can disrupt normal business operations. Furthermore, if the server responsible for reconciliation fails, manual recovery is required, further impacting efficiency and reliability. Therefore, a more reliable and efficient solution is needed. Summary of the Invention

[0004] To address the problems of existing technologies, this application provides a data verification method, system, device, and storage medium. The technical solution is as follows:

[0005] This application provides a data verification method, the method comprising:

[0006] Multiple data verification servers send verification task retrieval requests to the task relay server respectively;

[0007] The task relay server responds to the verification task acquisition request sent by each data verification server, and determines the verification task to be processed corresponding to each data verification server from multiple data verification tasks. The verification task to be processed includes verification scope information and data block parameters. The data block parameters are determined by the task control server based on database latency information when generating the data verification task.

[0008] The task relay server sends the pending verification task to the corresponding data verification server.

[0009] Each data verification server determines the target data to be verified based on the verification scope information in the corresponding verification task to be processed;

[0010] Each data verification server divides the target data to be verified into multiple data blocks based on the data block parameters of the corresponding verification task to be processed.

[0011] Each data verification server creates multiple data verification threads corresponding to the number of the multiple data blocks to be verified;

[0012] Each data verification server performs data verification on the multiple data blocks to be verified based on the multiple data verification threads.

[0013] This application also provides a data verification system, which includes a task relay server and multiple data verification servers;

[0014] The plurality of data verification servers are used to send verification task acquisition requests to the task relay server respectively; and to determine the target data to be verified based on the verification range information in the corresponding verification task to be processed; and to divide the target data to be verified into multiple data blocks to be verified based on the data block parameters of the corresponding verification task to be processed; and to create multiple data verification threads corresponding to the number of the multiple data blocks to be verified; and to perform data verification on the multiple data blocks to be verified based on the multiple data verification threads respectively.

[0015] The task relay server is used to respond to the verification task acquisition request sent by each data verification server, determine the verification task to be processed corresponding to each data verification server from multiple data verification tasks, the verification task to be processed includes verification scope information and data block parameters, the data block parameters are determined by the task control server based on database latency information when generating the data verification task; and send the verification task to be processed to the corresponding data verification server.

[0016] In another aspect, this application provides an apparatus comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data verification method described above.

[0017] In another aspect, this application provides a computer-readable storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the above-described data verification method.

[0018] The data verification method, system, equipment, and storage medium provided in this application have the following technical advantages:

[0019] In this application, multiple data verification servers send verification task retrieval requests to a task relay server. In response to each data verification server's request, the task relay server determines the pending verification task for each data verification server from among the multiple data verification tasks. The pending verification task includes verification scope information and data block parameters, which are determined by the task control server based on database latency information when generating the data verification task. Then, the task relay server sends the pending verification task to the corresponding data verification server. Next, each data verification server determines the target to be verified based on the verification scope information in its corresponding pending verification task. Each data verification server can divide the target data to be verified into multiple data blocks based on the data block parameters of the corresponding verification task. Then, multiple data verification threads corresponding to the number of data blocks to be verified are created. Based on the multiple data verification threads, data verification is performed on the multiple data blocks to be verified respectively. This can combine a task transfer server and multiple data verification servers, which can greatly improve the efficiency and flexibility of data verification. Moreover, the data block parameters can be determined in a timely manner based on the database latency information, and the number of data verification threads can be flexibly adjusted according to the data verification requirements and the actual situation of the database to control the concurrency and improve the security and stability of the database during data verification.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0023] Figure 2 This is a flowchart of a data verification method provided in an embodiment of this application;

[0024] Figure 3 This is a flowchart of another data verification method provided in the embodiments of this application;

[0025] Figure 4 This is a flowchart of another data verification method provided in the embodiments of this application;

[0026] Figure 5 This is a flowchart of another data verification method provided in the embodiments of this application;

[0027] Figure 6 This is a schematic diagram of the architecture of the data verification method provided in the embodiments of this application;

[0028] Figure 7 This is a flowchart of another data verification method provided in the embodiments of this application;

[0029] Figure 8 This is a schematic diagram of an architecture provided in an embodiment of this application when a data verification server malfunctions;

[0030] Figure 9 This is another schematic diagram of the data verification method provided in the embodiments of this application;

[0031] Figure 10 This is a schematic diagram of an architecture provided by an embodiment of the present application when each data verification server creates multiple data verification threads to perform data verification;

[0032] Figure 11 This is a flowchart of another data verification method provided in the embodiments of this application;

[0033] Figure 12 This is a flowchart of another data verification method provided in the embodiments of this application;

[0034] Figure 13 This is a schematic diagram of the structure of one embodiment of the data verification system provided in this application.

[0035] Figure 14 This is a hardware structure block diagram of a data verification server provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0038] Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and cryptographic algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying platform, a platform product service layer, and an application service layer.

[0039] The data verification method disclosed in this application involves data that can be stored on a blockchain, such as data to be verified.

[0040] Please see Figure 1 , Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application, such as... Figure 1 As shown, the application environment may include a task control server 01, a task transfer server 02, and multiple data verification servers 03.

[0041] In this embodiment, the task control server 01 can generate corresponding data verification tasks at preset intervals or at fixed recurring times within each cycle (e.g., daily). The preset intervals can be set according to actual application requirements. The generated data verification tasks are then sent to the task transfer server 02 for storage. Subsequently, multiple data verification servers 03 can retrieve the corresponding pending verification tasks from the task transfer server 02 and perform data verification accordingly. The task control server 01 can also obtain database latency information corresponding to historical data verification tasks to determine the data block parameters of the data verification tasks. This allows control over the number of threads used by the data verification servers when executing data verification tasks, improving concurrency while ensuring database security. This avoids the potential security risks (e.g., database crashes) caused by excessive concurrent requests, ensuring the normal operation of other services. In one embodiment, the task control server 01 can also monitor the task execution status information of each data verification task, thereby determining the concurrent access count of each database. If the concurrent access count of any database is greater than or equal to a preset threshold, the task control server 01 can temporarily store the data verification task associated with that database locally, i.e., suspend its distribution to the task transfer server 02, and wait until the concurrent access count of that database is less than the preset threshold before distributing it. The aforementioned preset threshold can be set based on concurrent test results and actual application requirements, enabling macro-level control of concurrency from the perspective of the task control server. This avoids potential security risks to the database load caused by excessive concurrent requests (such as database crashes) and ensures the normal operation of other services. The task control server 01 may include a network communication unit, a processor, and a memory, etc. In this embodiment, the task control server 01 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In some embodiments, the task control server 01 may include an electronic device with a processing unit, such as a laptop computer or desktop computer.

[0042] In this embodiment, the task relay server 02 can receive and store data verification tasks sent by the task control server 01, and respond to verification task retrieval requests sent by each data verification server to determine the pending verification tasks corresponding to each data verification server, so that each data verification server can perform data verification according to the corresponding pending verification tasks. Utilizing the task relay server 02 to store and relay data verification tasks can fully decouple the generation and execution processes of data verification tasks, ensuring that the task control server 01 and multiple data verification servers 03 do not interfere with each other, which is beneficial for improving concurrency and thus improving data verification efficiency. Specifically, the task relay server 02 may include a network communication unit, a processor, and a memory, etc. In this embodiment, the task relay server 02 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. In one specific embodiment, the task relay server 02 can be a message queue server, which includes multiple data verification tasks arranged in order of task generation time from earliest to latest. That is, each time a data verification task generated and sent by the task control server 01 is received, it will be stored at the tail of the message queue, so that the data verification task stored earlier can be executed first.

[0043] In this embodiment, multiple data verification servers 03 can be used to send verification task retrieval requests to the task relay server 02, thereby obtaining task execution data for verification from the task relay server 02. These multiple data verification servers operate independently and do not affect each other. After each data verification server completes its current verification task, it can retrieve a new task from the task relay server 02. Multiple data verification servers can perform data verification simultaneously, resulting in high concurrency and large data throughput, which can greatly improve data verification efficiency and flexibly achieve scaling up and down. Specifically, the data verification server can divide the data to be verified into multiple blocks based on the data block parameters of the corresponding verification task to be processed, and then create multiple data verification threads corresponding to the number of data blocks to be verified, and perform concurrent verification in multiple threads. Specifically, each data verification server 03 may include a network communication unit, a processor, and a memory, etc. In this embodiment, each data verification server 03 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0044] Please refer to Figure 1 In practical applications, task control server 01 generates corresponding data verification tasks and distributes them to task relay server 02 for storage. Multiple data verification servers 03 send verification task retrieval requests to task relay server 02. Task relay server 02 responds to each data verification server 03's request by determining the pending verification task for each data verification server 03 from the multiple data verification tasks. The pending verification task includes verification scope information and data block parameters, which are determined by task control server 01 based on database latency information when generating the data verification task. Then, task relay server 02 can send the pending verification task to the corresponding data verification server 03. Next, each data verification server 03 can divide the data to be verified into multiple blocks based on the corresponding pending verification task's data block parameters, and then create multiple data verification threads corresponding to the number of pending data blocks for concurrent verification.

[0045] In addition, it should be noted that, Figure 1This is merely a schematic diagram of an application environment provided by an embodiment of this application. In one embodiment, the application environment may further include at least two data storage systems (each data storage system may include a corresponding database). These at least two data storage systems utilize a data transmission scheme to store the same business data. For example, when two data storage systems are included, the first data storage system can be used for real-time business data management to support the normal operation of real-time business. Specifically, the first data storage system may be, for example, MySQL. The second data storage system may be a heterogeneous data source, used to achieve more massive and persistent data storage, suitable for near real-time data query and analysis scenarios. Specifically, the second data storage system may be, for example, Elasticsearch, HBase, MongoDB, etc. Due to the capacity limitation of the first data storage system, in order to avoid affecting real-time business, historical data will be cleaned up after a period of time to release available space. Therefore, a data transmission scheme can be used to transfer data from the first data storage system to the second data storage system for backup storage. Since there may be omissions or errors in data transmission, it is necessary to perform data verification to ensure that the total amount of data in the first data storage system and the second data storage system are consistent, and that the data content is consistent. Each data verification server needs to obtain the corresponding data to be verified from the above-mentioned at least two data storage systems when performing data verification. The aforementioned task control server 01, task relay server 02, and multiple data verification servers 03 can be directly or indirectly connected via wired or wireless communication, and this application is not limited thereto.

[0046] Figure 2 This is a flowchart illustrating a data verification method provided in an embodiment of this application. This specification provides the operational steps of the method described in the embodiments or flowchart, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiments or accompanying drawings. Specifically, as... Figure 2 As shown, the method may include:

[0047] S201: Multiple data verification servers send verification task retrieval requests to the task relay server respectively.

[0048] In this embodiment of the application, the aforementioned verification task acquisition request can be sent by each data verification server to the task relay server after the currently executing pending verification task is completed.

[0049] In practical applications, multiple data verification servers can execute their respective data verification tasks independently, without affecting each other. After completing a data verification task, each data verification server can send a verification task retrieval request to the task relay server to obtain a new data verification task and complete the data verification. Utilizing multiple independent data verification servers enables parallel execution of data verification tasks, greatly improving data verification efficiency. Furthermore, it allows for flexible scaling of the data verification process based on business needs and data volume. For example, as business needs and data volume increase, data verification efficiency can be rapidly improved by adding more data verification servers.

[0050] S203: The task relay server responds to the verification task retrieval request sent by each data verification server and determines the verification task to be processed corresponding to each of the above data verification servers from multiple data verification tasks.

[0051] In this embodiment, the aforementioned task relay server can be a message queue server, including multiple data verification tasks arranged in ascending order of task generation time, i.e., a task queue, which can be a type of message queue. In a specific embodiment, multiple data verification servers can send verification task retrieval requests to the task relay server at different times. The task relay server can respond to each verification task retrieval request sent by the data verification server, determine the data verification task with the earliest task generation time from the multiple data verification tasks (task queue) arranged in ascending order of task generation time, and use it as the verification task to be processed corresponding to that data verification server. This allows for priority processing of data verification tasks with earlier generation times, achieving reasonable and orderly data verification.

[0052] In one specific embodiment, the data verification tasks in the task relay server are pre-generated and sent by the task control server. The task relay server stores these tasks and acts as a buffer and connector, effectively decoupling the generation and execution processes of the data verification tasks, ensuring that the task control server and multiple data verification servers do not interfere with each other. During peak data periods, when the data verification servers cannot process a large number of tasks in a timely manner, these tasks are stored in the task queue of the task relay server, awaiting retrieval and execution by the data verification servers. This does not affect the task control server's ability to continue generating data verification tasks, thus improving concurrency and data verification efficiency. This ensures high availability for the task control server and allows for flexible adjustment of the number of data verification servers based on actual data verification needs, enhancing data verification flexibility and significantly improving operational efficiency.

[0053] In one specific embodiment, the aforementioned verification task to be processed may include verification scope information and data block parameters, which are determined by the task control server based on database latency information when generating the data verification task.

[0054] In one embodiment, see Figure 3 The above methods may also include:

[0055] S301: The task control server generates corresponding data verification tasks at preset intervals.

[0056] In the embodiments of this application, each generated data verification task may include verification range information. The aforementioned verification range information can characterize the data range that the corresponding data verification task needs to verify. Specifically, the aforementioned verification range information may include range information of at least one dimension. For example, the aforementioned range information of at least one dimension may include time range information and database range information.

[0057] The task control server generating corresponding data verification tasks at preset intervals may include: the task control server generating corresponding data verification tasks at preset intervals according to preset task division rules, and determining the verification scope information of the corresponding data verification task. The preset task division rules can be pre-configured based on actual application needs. Specifically, the task control server generating corresponding data verification tasks at preset intervals may include: the task control server generating corresponding data verification tasks at preset intervals, and determining the historical time period corresponding to the data verification task according to preset verification time period information. The preset time can represent the time interval between two adjacent data verification tasks generated by the task control server, and the preset verification time period information can represent the rule for determining the historical time period corresponding to the generated data verification task based on the current time; at this time, the verification scope information may include time range information. In a specific embodiment, the preset time can be 10 minutes, and the preset verification time period information can be from 30 minutes to 20 minutes before the current time. For example, data verification tasks can be generated at 10:00, 10:10, 10:20, etc. For the data verification task generated at 10:00, the corresponding verification range information can include 9:30 to 9:40. In practical applications, the preset time can also be 5 minutes, and the preset verification time period information can be from 30 minutes to 25 minutes before the current time. This application is not limited to this.

[0058] In this embodiment, besides the time dimension, the overall verification task can be further subdivided from other dimensions. For example, the verification scope information mentioned above can also include database scope information. The task control server generating corresponding data verification tasks at preset intervals can include: the task control server can generate multiple corresponding data verification tasks at preset intervals, determine the historical time period corresponding to each data verification task according to the preset verification time period information, and determine the database identification information corresponding to each data verification task according to the identification information of each database to be verified. In the above embodiment, multiple data verification tasks can be generated at 10:00, and the verification scope information corresponding to each data verification task can include "9:30~9:40, Database A", "9:30~9:40, Database B", "9:30~9:40, Database C", etc. In practical applications, the identification information of each database to be verified can be determined according to each MySQL instance to be verified. In other embodiments, the overall verification task can be further subdivided from other dimensions (besides the time period and database dimensions). For example, other dimensions can include the computer room, etc., and this application is not limited to this.

[0059] The task control server generates corresponding data verification tasks at preset intervals. Each data verification task can include verification scope information, which can include at least one dimension of scope information. This enables the task control server to perform overall task division in multiple dimensions (such as time and database), making the data verification tasks that each data verification server needs to complete each time relatively lightweight. This allows the distributed architecture including multiple data verification servers to perform data verification in parallel, which is beneficial to fully improve the concurrency of data verification, thereby improving the efficiency and throughput of data verification.

[0060] S303: The task control server retrieves historical data and verifies the database delay information corresponding to the task.

[0061] In this embodiment, the aforementioned historical data verification task can indicate the most recent completed data verification task, and the aforementioned database latency information can indicate the average database access latency during the execution of the historical data verification task. The task control server can obtain the execution results of each data verification task it generates. The execution results include the average database access latency during the execution of the historical data verification task, that is, the average response latency of the verification server accessing the database; the higher the average database access latency, the greater the access volume or number of requests handled by the database.

[0062] S305: When generating the corresponding data verification task, the task control server determines the data block parameters of the corresponding data verification task based on the database delay information.

[0063] In one specific embodiment, the above data chunking parameters may include the target chunk data volume, i.e., the size of the data chunk. Please refer to... Figure 4 When generating the corresponding data verification task, the task control server determines the data block parameters of the corresponding data verification task based on the database latency information, which may include:

[0064] S401: When the task control server generates the corresponding data verification task, it obtains the initial block data volume of the corresponding data verification task.

[0065] In this embodiment of the application, the initial block data volume can be the data block size set according to actual application needs, for example, 10,000 records; or it can be the target block data volume of the historical data verification task.

[0066] S403: When the task control server determines that the above database delay information meets the preset adjustment conditions, it adjusts the data volume according to the above initial block data volume to obtain the target block data volume of the corresponding data verification task.

[0067] Specifically, the aforementioned preset adjustment conditions may include: the aforementioned database delay information being greater than or equal to a first delay threshold; or: the aforementioned database delay information being less than or equal to a second delay threshold.

[0068] In one embodiment, when the task control server determines that the database latency information meets the preset adjustment conditions, adjusting the data volume according to the initial block data volume to obtain the target block data volume for the corresponding data verification task may include:

[0069] When the task control server determines that the above database delay information is greater than or equal to the first delay threshold, it performs data volume increase processing based on the above initial block data volume to obtain the target block data volume of the corresponding data verification task.

[0070] Specifically, increasing the data volume based on the initial data block size may include, but is not limited to: doubling the initial data block size; or; adding a first preset data volume to the initial data block size, wherein the first preset data volume can be determined based on actual application requirements and adjustment test results, for example, 10,000 data blocks.

[0071] Since the database needs to support various real-time front-end services in addition to providing services for data verification tasks, it must ensure secure and stable operation. When the database latency is greater than or equal to the first latency threshold, it indicates high database load, which may cause delays in other services or even database crashes. In this case, the data volume can be increased based on the initial block size. For example, if the initial block size is 10,000 records, and the total amount of data to be verified for the verification task is 100,000 records, the data verification server would normally create 10 threads for concurrent verification. However, by doubling the initial block size to 20,000 records, the data verification server will create 5 threads for concurrent verification when the total amount of data to be verified for the task is 100,000 records. This reduces database access concurrency, improves database security and stability, and ensures the normal operation of other services.

[0072] In another embodiment, when the task control server determines that the database latency information meets the preset adjustment conditions, the data volume adjustment is performed based on the initial block data volume to obtain the target block data volume for the corresponding data verification task, which may include:

[0073] When the task control server determines that the above database delay information is less than or equal to the second delay threshold, it performs data volume reduction processing based on the above initial block data volume to obtain the target block data volume of the corresponding data verification task.

[0074] Specifically, the data volume increase processing based on the initial block data volume may include, but is not limited to: reducing the initial block data volume by half; or; reducing the initial block data volume by a second preset data volume, wherein the second preset data volume can be determined in combination with actual application needs and adjustment test results, for example, 10,000 records.

[0075] When the database latency information is less than or equal to the second latency threshold, it indicates that the database load pressure is low. At this time, the data volume can be reduced based on the initial block data volume mentioned above. For example, if the original initial block data volume is 20,000 records, and the total amount of data to be verified for the verification task is 100,000 records, the original data verification server would create 5 threads for concurrent verification. However, the initial block data volume can be reduced by half, resulting in a target block data volume of 10,000 records. In this case, with the total amount of data to be verified for the verification task being 100,000 records, the data verification server will create 10 threads for concurrent verification. This can improve the concurrency of data verification while ensuring database concurrency safety, thereby improving data verification efficiency.

[0076] The task control server obtains the initial block data volume when generating the corresponding data verification task; and adjusts the data volume when the above database latency information meets the preset adjustment conditions to obtain the target block data volume of the data verification task. It can flexibly adjust the concurrency of data verification based on the database load, improve the data verification efficiency while ensuring the security and stability of the database, and thus ensure the normal operation of other business that requires data.

[0077] S307: The task control server sends the corresponding data verification task to the task relay server.

[0078] In this embodiment, after the task control server generates a data verification task, it sends it to the task relay server for storage in a task queue. Specifically, the task relay server may include multiple data verification tasks arranged in ascending order of their generation time. Each time the task control server generates a new data verification task, it sends it to the tail of the task queue stored on the task relay server, thus enabling the earlier-stored data verification tasks to be executed first, achieving a reasonable and orderly data verification. Furthermore, when generating the corresponding data verification task, the task control server can determine the data block parameters based on database latency information, thereby enabling flexible concurrency control.

[0079] S205: The task transfer server sends the pending verification task to the corresponding data verification server.

[0080] In this embodiment of the application, the task relay server responds to the verification task acquisition request sent by each data verification server. After determining the verification task to be processed corresponding to each data verification server from multiple data verification tasks, the task relay server can send the verification task to be processed to the corresponding data verification server to realize subsequent data verification.

[0081] In an optional embodiment, the task transfer server described above may include multiple task transfer partitions, and each task transfer partition may include multiple data verification tasks arranged in ascending order of task generation time. See also... Figure 5 The aforementioned task relay server, in response to the verification task retrieval request sent by each data verification server, determines the pending verification task corresponding to each data verification server from multiple data verification tasks, which may include:

[0082] S501: The task transfer server responds to the verification task acquisition request sent by each data verification server and determines the task transfer partition corresponding to each data verification server based on the preset mapping information.

[0083] Specifically, the aforementioned preset mapping information can characterize the association between the multiple task transfer partitions and the multiple data verification servers. A task transfer partition can only be associated with a single data verification server, meaning that for a single task transfer partition, only one data verification server can retrieve data verification tasks from it. Conversely, a data verification server can be associated with at least one task transfer partition. In the initial stage of practical application, the number of task transfer partitions is consistent with the number of data verification servers, meaning that there is a one-to-one correspondence between multiple task transfer partitions and multiple data verification servers. This allows each data verification server to consistently access its corresponding task transfer partition and retrieve tasks in a reasonable and orderly manner.

[0084] S503: The task transfer server determines the data verification task with the earliest task generation time from the task transfer partition corresponding to each data verification server.

[0085] In this embodiment, each task transfer partition may include multiple data verification tasks arranged in ascending order of task generation time. In a specific embodiment, the data verification tasks in each task transfer partition may be sent by the task control server through a round-robin method. When there are multiple task transfer partitions in the task transfer server, the task control server can send the generated data verification tasks to these multiple task transfer partitions in a round-robin manner, thereby achieving load balancing of data verification task allocation and ensuring that the data verification tasks generated first are picked up and executed by the corresponding data verification servers first, achieving reasonable and orderly data verification. Figure 6 As shown, Figure 6 This is a schematic diagram of the architecture of a data verification method provided in an embodiment of this application. In the task transfer server, the dashed boxes (e.g., 601) represent each task transfer partition, and the blocks containing numbers represent stored data verification tasks (e.g., 602). The numbers in the blocks represent the task generation time; the smaller the number, the earlier the task was generated. Figure 6 As can be seen, each task transfer partition includes multiple data verification tasks arranged in order of their generation time from earliest to latest. There are associations between multiple task transfer partitions and multiple data verification servers. Each data verification server can access the corresponding task transfer partition to obtain and execute the data verification tasks.

[0086] S505: The task transfer server will take the data verification task with the earliest task generation time as the pending verification task of the corresponding data verification server.

[0087] In this embodiment, after determining the data verification task with the earliest task generation time in the task transfer partition corresponding to each data verification server, the task transfer server can designate the data verification task with the earliest task generation time as the pending verification task for the corresponding data verification server. Please continue to refer to... Figure 6 For example, at this time, data verification task 1 can be used as the data verification task to be processed on data verification server 1; data verification task 2 can be used as the data verification task to be processed on data verification server 2; and data verification task 3 can be used as the data verification task to be processed on data verification server 3.

[0088] In practical applications, when multiple data verification servers send verification task retrieval requests to a task relay server, the task relay server needs to send the data at the head of the task queue to the corresponding data verification server sequentially according to the request time. This means that multiple data verification servers cannot concurrently access the task relay server to retrieve tasks. However, by utilizing multiple task relay partitions within the task relay server, each partition containing multiple data verification tasks arranged in chronological order of task generation, the task relay server can respond to each data verification server's verification task retrieval request by determining the corresponding task relay partition for each data verification server based on preset mapping information. This allows multiple data verification servers to simultaneously access the task relay server to retrieve data verification tasks, further improving data verification concurrency, efficiency, and throughput, and achieving a rational and orderly data verification process.

[0089] In practical applications, the actual operating environment of the data verification server is quite complex. Issues such as server crashes, network anomalies, thread errors, and memory exhaustion may occur, preventing the server from performing data verification tasks and requiring manual intervention, which is time-consuming and labor-intensive. Therefore, in this embodiment, please refer to... Figure 7 The above methods may also include:

[0090] S701: The task relay server monitors the operating status of the above-mentioned multiple data verification servers.

[0091] In one specific embodiment, the task relay server can monitor the operating status of the above-mentioned multiple data verification servers based on a heartbeat mechanism. Specifically, each data verification server can send a liveness message to the task relay server at preset time intervals. When the task relay server detects that the time waiting for any data verification server to send a liveness message exceeds a preset time threshold, it determines that the data verification server has an abnormal operating status.

[0092] S703: If the task transfer server detects that any data verification server has an abnormal operating status, the target task transfer partition corresponding to the abnormal data verification server is determined based on the preset mapping information.

[0093] S705: The task relay server determines the target data verification server from the plurality of data verification servers.

[0094] Specifically, the target data verification server is a data verification server that is running normally. In this embodiment, the target data verification server can be any one of the plurality of data verification servers that is running normally. In an optional embodiment, the task relay server can be a Kafka server, and the plurality of data verification servers can be in the same Kafka consumer group, thereby enabling the determination of the target data verification server from the data verification servers in the same group.

[0095] S707: The task transfer server establishes the association between the target data verification server and the target task transfer partition.

[0096] S709: The task transfer server updates the preset mapping information based on the association between the target data verification server and the target task transfer partition.

[0097] In this embodiment, the task transfer server can establish an association between the target data verification server and the target task transfer partition. Based on this association, it updates the preset mapping information, allowing the target data verification server to subsequently take over and execute data verification tasks within the target task transfer partition. Please refer to... Figure 8 Suppose that data verification server 1 fails at this time. The task relay server determines data verification server 2 as the target data verification server from among the multiple data verification servers. Then, an association can be established between data verification server 2 and the partition originally handled by data verification server 1, thereby updating the original preset mapping information. That is, data verification server 2 can take over the partition originally handled by data verification server 1, and subsequently, data verification server 2 can take turns receiving and executing data verification tasks from its two corresponding task relay partitions.

[0098] By monitoring the operational status of the multiple data verification servers through the task relay server, if the task relay server detects an abnormal operational status of any data verification server, other data verification servers can take over the partition corresponding to the abnormal data verification server. This enables automated failover without affecting the execution of data verification tasks, which helps improve the overall availability of the system, thereby enhancing the reliability and stability of data verification and avoiding adverse impacts on normal business operations.

[0099] S207: Each data verification server determines the target data to be verified based on the verification scope information in the corresponding verification task to be processed.

[0100] In this embodiment, the target data to be verified may include corresponding business data in at least two data storage systems, which store the same business data using a data transmission scheme. Specifically, the business data may be business information generated through preset operations, and each piece of business data may include a data primary key and the values ​​of multiple data fields. For example, business data may include transaction order data, where the data primary key of each transaction order may be a unique order number, and the data fields may include, but are not limited to, order time, user ID, payment amount, and receiving merchant.

[0101] In this embodiment, when two data storage systems are included, the first data storage system can be used for real-time business data management to support the normal operation of real-time business (e.g., supporting the aforementioned real-time transaction business). Specifically, the first data storage system can be, for example, MySQL. In a specific embodiment, the second data storage system can be a heterogeneous data source to achieve more massive and persistent data storage, suitable for near real-time data querying and analysis scenarios. Specifically, the second data storage system can be, for example, Elasticsearch, HBase, MongoDB, etc. Generally, due to the capacity limitations of the first data storage system, to avoid impacting real-time business, historical data is cleaned up after a period of time to release available space. Therefore, a data transmission scheme (data synchronization operation) can be used to transfer data from the first data storage system to the second data storage system for backup storage for subsequent querying and other applications, such as historical order querying and refunds. At this time, the data in MySQL has been cleared, and the stored data needs to be queried in Elasticsearch. However, since there may be omissions or errors in the data transmission process, it is necessary to verify the data before and after transmission to ensure that the total amount of data in the first and second data storage systems is consistent, and the data content is consistent, thereby providing reliable data support for subsequent data querying and other operations. When performing data verification, each data verification server can obtain the target data to be verified based on the verification scope information in the corresponding verification task. The target data to be verified can include two parts: the business data in the first data storage system that corresponds to the above-mentioned verification scope information, and the business data in the second data storage system that corresponds to the above-mentioned verification scope information.

[0102] In an optional embodiment, the method further includes a step of pre-synchronizing business data in the first data storage system to obtain business data in the second data storage system. Specifically, this step may include: collecting incremental logs from the first data storage system, storing them in a message queue via a relay, and parsing the incremental logs in the second data storage system to synchronously write the business data. Specifically, the incremental logs can represent database change information; in practical applications, these incremental logs can be MySQL's binlog.

[0103] In this embodiment, since each verification task includes verification scope information, each data verification server obtains the target data to be verified based on the verification scope information in the corresponding verification task. For example, the verification scope information of a verification task may include time range information and database range information, such as "9:30~9:40, database A". Each piece of business data includes data attribute information, such as storage time and storage location (the database identifier). Therefore, the corresponding business data can be filtered out using the aforementioned verification scope information in the first data storage system and the second data storage system, respectively, as the aforementioned target data to be verified. Please refer to... Figure 9 , Figure 9 This diagram illustrates a data verification architecture provided in an embodiment of this application. It allows for the rapid and accurate determination of the data corresponding to a task, thereby improving the reliability of data verification.

[0104] S209: Each data verification server divides the target data to be verified into multiple data blocks based on the data block parameters of the corresponding verification task to be processed.

[0105] Specifically, the above data segmentation parameters may include the target segmentation data volume. Each data verification server, based on the verification range information in the corresponding verification task to be processed, determines the target data to be verified and the total amount of the target data to be verified. For example, the total amount of the target data to be verified is 100,000 records.

[0106] In this embodiment, each data verification server, based on the data block parameters of the corresponding verification task to be processed, divides the target data to be verified into multiple data blocks to be verified. This may include: each data verification server determining the total amount of the target data to be verified, and dividing the target data to be verified into multiple data blocks according to the total amount of data and the target block data size. For example, if the total amount of the target data to be verified is 100,000 records and the target block data size is 10,000 records, then 10 data blocks to be verified are obtained after data block division.

[0107] S211: Each data verification server creates multiple data verification threads corresponding to the number of the multiple data blocks to be verified.

[0108] In this embodiment of the application, the number of data verification threads created by the data verification server is the same as the number of data blocks to be verified.

[0109] S213: Each data verification server performs data verification on the multiple data blocks to be verified based on the above multiple data verification threads.

[0110] In this embodiment of the application, after the data verification server creates a data verification thread that matches the number of data blocks to be verified, it can perform data verification on a database to be verified in each data verification thread.

[0111] like Figure 10 As shown, Figure 10 This is a schematic diagram of an architecture provided by an embodiment of the present application when each data verification server creates multiple data verification threads to perform data verification. The short line after each data verification server (e.g., 1001) in the diagram represents a data verification thread. To avoid too many lines, only the process of thread 1001 obtaining data from two data storage systems is shown here. In actual applications, each data verification thread can pull data from two data storage systems simultaneously.

[0112] When generating data verification tasks through the task control server, the data block parameters are determined based on database latency information. This allows for flexible adjustment of the data verification concurrency based on database load. Then, during the actual data verification on the data verification server, a corresponding number of threads are created based on the data block parameters. Multiple data verification threads are used to achieve parallel processing, which can improve the data verification concurrency while ensuring database security and stability, thereby improving data verification efficiency, avoiding waste of verification resources, and ensuring the normal operation of other business processes that require data verification.

[0113] In one embodiment, the number of data verification threads can be set according to actual application requirements. After each data verification server determines the target data to be verified based on the verification range information in the corresponding verification task to be processed, the above method may further include:

[0114] 1) Each data verification server creates a preset number of data verification threads;

[0115] Specifically, the aforementioned preset quantity can be set according to actual application needs. In one embodiment, the aforementioned preset quantity is consistent with the number of data tables in the target data to be verified.

[0116] 2) Each data verification server determines the corresponding data block to be verified based on each data verification thread;

[0117] 3) In each data verification thread, each data verification server re-slices the corresponding data block to be verified according to the above data block parameters to obtain multiple sub-data blocks;

[0118] 4) Each data verification thread creates multiple sub-threads, and each sub-thread performs data verification on the corresponding sub-data block.

[0119] In some scenarios, the number of data verification threads is sometimes set to a fixed value based on actual application requirements, or a value consistent with the number of data tables. In this case, after each data verification thread determines the corresponding data block to be verified (e.g., data from a certain data table), the corresponding data block to be verified can be re-sliced ​​using data partitioning parameters to obtain multiple sub-data blocks, which also allows for flexible adjustment of concurrency. This can improve data verification concurrency while ensuring database security and stability, thereby improving data verification efficiency, avoiding waste of verification resources, and ensuring the normal operation of other business processes that require data integration.

[0120] In one specific embodiment, the target data to be verified may include a first data to be verified and a second data to be verified. The second data to be verified is obtained by pre-synchronizing the first data to be verified. In this embodiment, the first data to be verified may be business data corresponding to the verification scope information in the first data storage system, and the second data to be verified may be business data corresponding to the verification scope information in the second data storage system. That is, the second data to be verified is backup data corresponding to the first data to be verified. After dividing the target data to be verified into blocks, multiple data blocks to be verified are obtained. Each data block to be verified also includes the first data to be verified and the second data to be verified. The second data to be verified is obtained by pre-synchronizing the first data to be verified. The first data to be verified in the data block to be verified is business data corresponding to a portion of the verification scope information, and the second data to be verified may be business data corresponding to the verification scope information in the second data storage system (the second data to be verified is backup data corresponding to the first data to be verified).

[0121] In one specific embodiment, please refer to Figure 11 Each of the aforementioned data verification servers, based on the aforementioned multiple data verification threads, performs data verification on multiple data blocks to be verified, which may include:

[0122] S1101: Each data verification server, in each data verification thread, determines the data identification information of each piece of data in the first data to be verified.

[0123] Specifically, each of the above data entries can be business data, and the data identification information can represent a unique identifier that distinguishes this data entry from other data. Specifically, the data identification information can be a primary key, such as a unique order number.

[0124] S1103: In each data verification thread, each data verification server determines the associated data of each data from the second data to be verified based on the data identification information of each data.

[0125] Specifically, based on the data identifier information of each data item in the first set of data to be verified, the associated data of each data item in the second set of data to be verified can be determined. The associated data of each data item can be data in the second set of data that has the same data identifier information as that data item. Ideally, each data item in the first set of data to be verified should have exactly the same associated data in the second set of data to be verified.

[0126] In this embodiment, for any piece of data in the first set of data to be verified, if there is no data with the same data identifier information in the second set of data to be verified (i.e., no associated data exists), that piece of data in the first set of data to be verified can be recorded to generate a verification anomaly log. In practical applications, if there is no data with the same data identifier information in the second set of data to be verified, that is, data that exists in the first data storage system but not in the second data storage system, it may be that a data synchronization operation has resulted in a missed transmission. This can be recorded and the data can be retransmitted to improve the reliability of data synchronization.

[0127] S1105: Each data verification server performs data field comparison on each piece of data and its associated data in each data verification thread.

[0128] In an optional embodiment, the second data to be verified can be data from a heterogeneous data source (i.e., the second data storage system can be a heterogeneous data source), and it is necessary to determine the consistency of the data by comparing each field. Specifically, after determining each piece of data and its associated data, data field comparison can be performed. This can include aligning the data field types and then performing a consistency comparison on the corresponding field values. For example, for a transaction order data, the data fields may include, but are not limited to, order time, user ID, payment amount, and receiving merchant. For each piece of data in the first data to be verified and its associated data in the second data to be verified, the order time, user ID, payment amount, and receiving merchant can be checked one by one to ensure that the data content before and after transmission is the same.

[0129] In this embodiment, for any piece of data in the first set of data to be verified, if a discrepancy is detected between any piece of data and its associated data (the values ​​of any corresponding fields are different), that piece of data in the first set of data to be verified can be recorded, generating a verification exception log. In practical applications, when a discrepancy exists, it may be due to a mistransmission during data synchronization. This can be recorded and the data retransmitted, thereby improving the reliability of data synchronization and providing more reliable data support for subsequent data use (such as data querying, historical order refunds, etc.).

[0130] In this embodiment of the application, by having each data verification server compare the data fields of each data entry and its associated data, data verification can be performed more meticulously and reliably, thereby improving the reliability of data verification.

[0131] In one specific embodiment, each data verification task includes task-associated database information. Specifically, this task-associated database information can characterize the database identifier information that the data verification task needs to access. In an optional embodiment, the task-associated database information can be the database range information in the corresponding verification range information. Furthermore, the task control server generates corresponding data verification tasks every preset time interval. Please refer to [link to relevant documentation]. Figure 12 The above methods may also include:

[0132] S1201: The task control server monitors the task execution status information of multiple data verification tasks.

[0133] Specifically, the task execution status information mentioned above can indicate whether the corresponding data verification task is in progress. The task execution status information mentioned above can specifically include in progress, not executed, executed, etc.

[0134] S1203: The task control server checks the task execution status information and task-related database information of the multiple data verifications to determine the number of concurrent accesses to the multiple databases.

[0135] In this embodiment of the application, the task control server determines the number of concurrent accesses to multiple databases based on the task execution status information and task-associated database information of the multiple data verification tasks. This can include: the task control server determines the number of tasks in execution corresponding to each database based on the task execution status information and task-associated database information of the multiple data verification tasks, and uses the number of tasks in execution as the number of concurrent accesses to that database.

[0136] S1205: If the task control server detects that the number of concurrent accesses to any database is greater than or equal to a preset threshold, the data verification task associated with the database whose number of concurrent accesses is greater than or equal to the preset threshold is stored locally.

[0137] In this embodiment, the aforementioned preset threshold can be set based on actual access test results and application requirements. For example, the preset threshold can be five. When the task control server detects that the concurrent access count of any database is greater than or equal to the preset threshold, it indicates a high database system load, potentially leading to database crashes and affecting normal business operations. Therefore, concurrency control can be implemented from the task control server side. Data verification tasks associated with databases whose concurrent access count is greater than or equal to the preset threshold are stored locally and not sent out. When the concurrent access count of that database is detected to be less than the preset threshold, the locally stored data verification tasks associated with that database are sent to the task transfer server for further execution. This approach facilitates macro-level concurrency control, provides greater adjustment leverage, quickly alleviates database load pressure, prevents database system crashes due to high load, ensures business security, and achieves the highest verification performance within the database system's performance safety limits, thereby improving the security and reliability of data verification.

[0138] In the embodiments of this specification, a task relay server is used to store the data verification tasks generated by the task control server and to relay the data verification tasks, playing a buffering and connecting role. This fully decouples the generation and execution processes of the data verification tasks, ensuring that the task control server and multiple data verification servers do not interfere with each other. Using multiple independent data verification servers enables parallel execution of data verification tasks, greatly improving data verification efficiency and allowing for flexible scaling of the data verification business according to business needs and data volume. When generating the corresponding data verification task, the task control server can determine the data block parameters based on database latency information, thereby enabling flexible concurrency control. By obtaining the initial block data volume when generating the corresponding data verification task and adjusting the data volume when the aforementioned database latency information meets preset adjustment conditions, the target block data volume of the data verification task can be obtained. This allows for flexible adjustment of the data verification concurrency based on database load, improving data verification efficiency while ensuring database security and stability, and thus ensuring the normal operation of other business processes that require data integration. Based on preset mapping information, the task transfer partition corresponding to each data verification server is determined. This allows multiple data verification servers to simultaneously access the task transfer server to obtain data verification tasks, which helps to further improve data verification concurrency, efficiency, and throughput, achieving reasonable and orderly data verification. By monitoring the operational status of the multiple data verification servers through the task transfer server, if the task transfer server detects an abnormal operational status of any data verification server, other data verification servers can take over the partition corresponding to the abnormal data verification server. This enables automated failover, without affecting the execution of data verification tasks, improving overall system availability, and thus enhancing the reliability and stability of data verification, avoiding adverse impacts on normal business operations. By creating multiple data verification threads for each data verification server, dividing the verification tasks to be processed, and enabling parallel processing through multiple data verification threads, the concurrency of data verification is further improved, thereby increasing data verification efficiency.

[0139] This application also provides an embodiment of a data verification system, such as... Figure 13 As shown, the system may include:

[0140] Task relay server 1310 and multiple data verification servers 1320;

[0141] The plurality of data verification servers 1320 are configured to send verification task acquisition requests to the task relay server respectively; determine target data to be verified based on the verification range information in the corresponding verification task to be processed; divide the target data to be verified into multiple data blocks based on the data block parameters of the corresponding verification task to be processed; create multiple data verification threads corresponding to the number of the multiple data blocks to be verified; and perform data verification on the multiple data blocks to be verified based on the multiple data verification threads respectively.

[0142] The task relay server 1310 is used to respond to the verification task acquisition request sent by each data verification server, determine the verification task to be processed corresponding to each data verification server from multiple data verification tasks, the verification task to be processed includes verification scope information and data block parameters, the data block parameters are determined by the task control server based on database latency information when generating the data verification task; and send the verification task to be processed to the corresponding data verification server.

[0143] In one specific embodiment, the data verification system further includes a task control server, which can be used to generate corresponding data verification tasks at preset intervals.

[0144] And obtain database delay information corresponding to historical data verification tasks;

[0145] And when generating the corresponding data verification task, the data block parameters of the corresponding data verification task are determined based on the database delay information;

[0146] And send the corresponding data verification task to the task relay server.

[0147] In one specific embodiment, the data block parameters include the target block data volume, and the task control server can also be used for:

[0148] When generating the corresponding data verification task, the initial block data volume of the corresponding data verification task is obtained;

[0149] And when it is determined that the database delay information meets the preset adjustment conditions, the data volume is adjusted according to the initial block data volume to obtain the target block data volume of the corresponding data verification task.

[0150] In one specific embodiment, the task transfer server includes multiple task transfer partitions, each task transfer partition including multiple data verification tasks arranged in ascending order of task generation time; the task transfer server 1310 can be used for:

[0151] In response to a verification task retrieval request sent by each data verification server, a task transfer partition corresponding to each data verification server is determined based on preset mapping information, wherein the preset mapping information characterizes the association between the multiple task transfer partitions and the multiple data verification servers.

[0152] And determine the data verification task with the earliest task generation time from the task transfer partition corresponding to each data verification server;

[0153] The data verification task with the earliest task generation time is designated as the pending verification task of the corresponding data verification server.

[0154] In one specific embodiment, the task relay server 1310 can also be used for:

[0155] Monitor the operating status of the multiple data verification servers;

[0156] And if any data verification server is found to have an abnormal operating status, the target task transfer partition corresponding to the abnormal data verification server is determined based on the preset mapping information.

[0157] And from the plurality of data verification servers, a target data verification server is determined, wherein the target data verification server is a data verification server in normal operating condition;

[0158] And establish the association between the target data verification server and the target task transfer partition;

[0159] And based on the association between the target data verification server and the target task transfer partition, update the preset mapping information.

[0160] In one specific embodiment, each data block to be verified includes a first data block to be verified and a second data block to be verified, wherein the second data block to be verified is obtained by pre-synchronizing the first data block to be verified; each data verification server 1320 can be used for:

[0161] In each data verification thread, determine the data identification information of each piece of data in the first data to be verified;

[0162] And in each data verification thread, based on the data identification information of each data, the associated data of each data is determined from the second data to be verified;

[0163] In each data verification thread, data fields are compared between each piece of data and its associated data.

[0164] In one specific embodiment, the task control server can be used to generate corresponding data verification tasks at preset time intervals, each data verification task including task-related database information. The task control server can also be used for:

[0165] Monitor the task execution status information of multiple data verification tasks;

[0166] And based on the task execution status information and task-related database information of the multiple data verification tasks, determine the number of concurrent accesses to multiple databases;

[0167] And if the number of concurrent accesses to any database is detected to be greater than or equal to a preset threshold, the data verification task associated with the database whose number of concurrent accesses is greater than or equal to the preset threshold is stored locally.

[0168] The components and method embodiments described in the system embodiments are based on the same application concept.

[0169] This application provides a computer device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the data verification method provided in the above method embodiments.

[0170] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0171] The methods and embodiments provided in this application can be executed in a mobile terminal, computer terminal, server, or similar computing device; that is, the aforementioned computer device may include a mobile terminal, computer terminal, server, or similar computing device. Taking running on a server as an example... Figure 14 This is a hardware structure block diagram of a server for a data verification method provided in an embodiment of this application. For example... Figure 14As shown, the server 1400 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1410 (CPUs 1410 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 1430 for storing data, and one or more storage media 1420 (e.g., one or more mass storage devices) for storing application programs 1423 or data 1422. The memory 1430 and storage media 1420 may be temporary or persistent storage. The program stored in the storage media 1420 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 1410 may be configured to communicate with the storage media 1420 and execute the series of instruction operations stored in the storage media 1420 on the server 1400. Server 1400 may also include one or more power supplies 1460, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1440, and / or one or more operating systems 1421, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0172] The input / output interface 1440 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1400. In one example, the input / output interface 1440 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In one example, the input / output interface 1440 can be a radio frequency (RF) module for wireless communication with the Internet.

[0173] Those skilled in the art will understand that Figure 14 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1400 may also include... Figure 14 The more or fewer components shown, or having the same Figure 14 The different configurations shown.

[0174] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a data verification method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data verification method provided in the above method embodiments.

[0175] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0176] Embodiments of this application also provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0177] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0178] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0179] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0180] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data verification method, characterized in that, The method includes: The task control server generates a data verification task and sends the data verification task to the task relay server; Multiple data verification servers send verification task retrieval requests to the task relay server respectively; The task relay server responds to the verification task acquisition request sent by each data verification server, and determines the verification task to be processed corresponding to each data verification server from multiple data verification tasks. The verification task to be processed includes verification scope information and data block parameters. The data block parameters are determined by the task control server based on database latency information when generating the data verification task. The task relay server sends the pending verification task to the corresponding data verification server. Each data verification server obtains the target data to be verified based on the verification scope information in the corresponding verification task to be processed; Each data verification server divides the target data to be verified into multiple data blocks based on the data block parameters of the corresponding verification task to be processed. Each data verification server creates multiple data verification threads corresponding to the number of the multiple data blocks to be verified; Each data verification server performs data verification on the multiple data blocks to be verified based on the multiple data verification threads.

2. The method according to claim 1, characterized in that, The task control server generates corresponding data verification tasks at preset intervals, and the method further includes: The task control server obtains historical data to verify the database delay information corresponding to the task; When generating a corresponding data verification task, the task control server determines the data block parameters of the corresponding data verification task based on the database delay information. The task control server sends the corresponding data verification task to the task relay server.

3. The method according to claim 2, characterized in that, The data block parameters include the target block data volume. When generating the corresponding data verification task, the task control server determines the data block parameters of the corresponding data verification task based on the database latency information, including: When generating a corresponding data verification task, the task control server obtains the initial block data volume of the corresponding data verification task. When the task control server determines that the database delay information meets the preset adjustment conditions, it adjusts the data volume according to the initial block data volume to obtain the target block data volume of the corresponding data verification task.

4. The method according to claim 1, characterized in that, The task relay server includes multiple task relay partitions, and each task relay partition includes multiple data verification tasks arranged in ascending order of task generation time. In response to a verification task retrieval request sent by each data verification server, the task relay server determines the pending verification tasks corresponding to each data verification server from the multiple data verification tasks, including: The task transfer server responds to the verification task acquisition request sent by each data verification server, and determines the task transfer partition corresponding to each data verification server based on preset mapping information. The preset mapping information represents the association relationship between the multiple task transfer partitions and the multiple data verification servers. The task relay server determines the data verification task with the earliest task generation time from the task relay partition corresponding to each data verification server; The task relay server will designate the data verification task with the earliest task generation time as the pending verification task of the corresponding data verification server.

5. The method according to claim 4, characterized in that, The method further includes: The task relay server monitors the operating status of the multiple data verification servers; If the task transfer server detects that any data verification server has an abnormal operating status, the target task transfer partition corresponding to the abnormal data verification server is determined based on the preset mapping information. The task relay server determines the target data verification server from the plurality of data verification servers, and the target data verification server is a data verification server in normal operating status; The task transfer server establishes an association between the target data verification server and the target task transfer partition; The task transfer server updates the preset mapping information based on the association between the target data verification server and the target task transfer partition.

6. The method according to claim 1, characterized in that, Each data block to be checked includes a first data block to be checked and a second data block to be checked, wherein the second data block to be checked is obtained by performing a data synchronization operation on the first data block to be checked in advance. Each data verification server, based on the plurality of data verification threads, performs data verification on the plurality of data blocks to be verified, including: Each data verification server, in each data verification thread, determines the data identification information of each piece of data in the first data to be verified. In each data verification thread, each data verification server determines the associated data of each data item from the second data to be verified based on the data identification information of each data item. Each data verification server, in each data verification thread, performs data field comparison on each data entry and its associated data.

7. The method according to claim 1, characterized in that, The task control server generates corresponding data verification tasks at preset intervals. Each data verification task includes task-related database information. The method further includes: The task control server monitors multiple data verification tasks to check their execution status information. The task control server determines the number of concurrent accesses to multiple databases by verifying the task execution status information and task-related database information of the multiple data checkers. If the task control server detects that the number of concurrent accesses to any database is greater than or equal to a preset threshold, the data verification task associated with the database whose number of concurrent accesses is greater than or equal to the preset threshold will be stored locally.

8. A data verification system, characterized in that, The system includes a task control server, a task relay server, and multiple data verification servers; The task control server is used to generate data verification tasks and send the data verification tasks to the task relay server; The plurality of data verification servers are used to send verification task acquisition requests to the task transfer server respectively; and to acquire target data to be verified based on the verification range information in the corresponding verification task to be processed. And based on the data block parameters of the corresponding verification task to be processed, the target data to be verified is divided into data blocks to obtain multiple data blocks to be verified. And create multiple data verification threads corresponding to the number of the multiple data blocks to be verified; And based on the multiple data verification threads, data verification is performed on the multiple data blocks to be verified respectively; The task relay server is used to respond to the verification task acquisition request sent by each data verification server, determine the verification task to be processed corresponding to each data verification server from multiple data verification tasks, the verification task to be processed includes verification scope information and data block parameters, the data block parameters are determined by the task control server based on database latency information when generating the data verification task; and send the verification task to be processed to the corresponding data verification server.

9. A data verification device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data verification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the data verification method as described in any one of claims 1 to 7.

Images