Data processing method and device, computer readable medium and electronic equipment

Abstract

The application belongs to the field of data processing, and relates to a data processing method, device, computer readable medium and electronic equipment. The method comprises the following steps: in response to a data checking request, a first data copy corresponding to a source node and a second data copy corresponding to a destination node are obtained, the first data copy comprises first primary key information and a first version number corresponding to each piece of data in the source node, and the second data copy comprises second primary key information and a second version number corresponding to each piece of data in the destination node; the first primary key information and the first version number are compared with the second primary key information and the second version number respectively to obtain a data checking result, and a target operation is performed according to the data checking result. The application can realize system lock-free data checking, reduces the occupation of system resources, and improves the efficiency of data checking.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and specifically relates to a data processing method, a data processing device, a computer-readable medium, and an electronic device. Background Technology

[0002] For an online business system, a large amount of data is constantly being transferred between various nodes. During this data transfer, middleware such as Kafka and Redis are used. If an anomaly occurs in any of these links, inconsistencies between the source and destination data will inevitably affect the system's operation. Therefore, data verification between nodes is essential. However, the data in the system is constantly changing, making data verification in a dynamic context a significant challenge.

[0003] Currently, data verification is mainly performed using locking mechanisms. First, a verification request is sent to the source end that wants to verify the data. Upon receiving the request, the source end pauses receiving new data from upstream, transmits existing data to the destination end, and then returns. Data is then retrieved from both the source and destination ends for verification. Finally, any discrepancies are sent to the destination end to complete the verification. For systems with large amounts of data, this method consumes significant resources due to the pause in data transmission for verification, impacting normal data transmission and consequently affecting the smooth operation of user services. Summary of the Invention

[0004] This application provides a data processing method, a data processing apparatus, a computer-readable medium, and an electronic device, which can overcome the problem in related technologies that data verification requires locking and pausing data transmission when system data is dynamically changing.

[0005] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0006] In a first aspect, embodiments of this application provide a data processing method, the method comprising: responding to a data verification request; obtaining a first data copy corresponding to a source node and a second data copy corresponding to a destination node, wherein the first data copy contains first primary key information and a first version number corresponding to each piece of data in the source node, and the second data copy contains second primary key information and a second version number corresponding to each piece of data in the destination node; comparing the first primary key information and the first version number with the second primary key information and the second version number respectively to obtain a data verification result, and performing a target operation based on the data verification result.

[0007] Secondly, embodiments of this application provide a data processing apparatus, comprising: a copy acquisition module, configured to respond to a data verification request and acquire a first data copy corresponding to a source node and a second data copy corresponding to a destination node, wherein the first data copy includes first primary key information and a first version number corresponding to each piece of data in the source node, and the second data copy includes second primary key information and a second version number corresponding to each piece of data in the destination node; and a data comparison module, configured to compare the first primary key information and the first version number with the second primary key information and the second version number respectively to obtain a data verification result, and perform a target operation based on the data verification result.

[0008] Thirdly, embodiments of this application provide a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in the above technical solutions.

[0009] Fourthly, embodiments of this application provide an electronic device, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform a data processing method as described in the above technical solutions by executing the executable instructions.

[0010] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the aforementioned data processing method.

[0011] The data processing method provided in this application can, on the one hand, obtain a first data copy corresponding to the source node and a second data copy corresponding to the destination node after responding to a data verification request. By comparing the first primary key information and the first version number in the first data copy with the second primary key information and the second version number in the second data copy, it can be determined whether the data sent by the source node and the data received by the destination node are the same. Since data copies are used, it is not necessary to stop data transmission or lock the data transmission line, thus realizing lock-free data verification and reducing the occupation of system resources. On the other hand, since this application only forms the first data copy and the second data copy based on the primary key information and version number in each row of data, the amount of data used in data verification is reduced, and the efficiency of data verification is improved.

[0012] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0013] Figure 1 An architectural block diagram of a system applying the technical solution of this application is shown schematically in one embodiment.

[0014] Figure 2 This diagram illustrates a process for data verification using a locking mechanism in related technologies.

[0015] Figure 3 A schematic diagram illustrating the steps of a data processing method in one embodiment is shown.

[0016] Figure 4 A schematic diagram illustrating the process of obtaining a first data copy and a second data copy in one embodiment is shown.

[0017] Figure 5 A schematic diagram illustrating the process of obtaining data verification results in one embodiment is shown.

[0018] Figure 6 The diagram illustrates a flowchart of an embodiment in which a target operation is performed based on data verification results.

[0019] Figure 7 The diagram illustrates a Redis-based distributed lock interface in one embodiment.

[0020] Figure 8 The diagram illustrates a workflow for data verification and data repair in one embodiment.

[0021] Figure 9 A schematic block diagram of a data processing apparatus in one embodiment is shown.

[0022] Figure 10 The diagram illustrates a computer system architecture block diagram suitable for implementing an electronic device according to an embodiment of the present application. Detailed Implementation

[0023] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0024] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0025] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0026] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0027] To facilitate understanding of the technical solutions in this application, the technical terms involved in this application are explained herein.

[0028] 1. Multiversion concurrency control (MCC or MVCC) is a commonly used concurrency control mechanism in database management systems and is also used in programming languages ​​to implement transactional memory. MVCC aims to solve the problems of multiple, long-running read operations and dead write operations caused by read-write locks. Each data item read by a transaction is a historical snapshot, depending on the implemented isolation level. Write operations do not overwrite existing data items but create a new version, which remains unchanged until all operations are committed. Snapshot isolation allows transactions to see the data state at the time it started.

[0029] MVCC primarily addresses the following issues:

[0030] 1. Ensuring that reads and writes do not block each other—that is, reads do not block writes, and writes do not block reads—improves transaction concurrency. Evolutionary approaches to enhancing concurrency include: ordinary locks, which allow only serial execution; read-write locks, which enable concurrent read and write operations; and multi-version concurrency control (MLC), which also enables concurrent read and write operations.

[0031] 2. Reduced probability of deadlock: MVCC uses optimistic locking, which does not require locking when reading data, and only locks necessary data rows for write operations.

[0032] 3. It solves the problem of consistent reads. Consistent reads are also known as snapshot reads. When querying a snapshot of the database at a certain point in time, you can only see the results of transactions committed and updated before that point in time, and you cannot see the results of transactions committed and updated after that point in time.

[0033] 2. Data Replication: Consistent reading is performed on each node of data flow in the system to generate snapshots, and data is generated by extracting primary key information and version number from the snapshots.

[0034] 3. Primary key information: Information that can distinguish each data row, such as node number, node name, etc.

[0035] 4. Version number: Used to indicate the update status of data in a data row. This version number is a globally incrementing version number. The version number will increment automatically when any one or more data in the data row changes.

[0036] 5. Source node and destination node: Determined according to the data flow direction. Among two adjacent nodes, the node that sends data is the source node, and the node that receives data is the destination node.

[0037] Figure 1 An exemplary system architecture block diagram illustrating the application of the technical solution of this application is shown schematically.

[0038] like Figure 1 As shown, the system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may include various electronic devices such as smartphones, tablets, laptops, desktop computers, smart TVs, and smart vehicle terminals. Server 103 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Network 102 is a communication medium of various connection types that can provide a communication link between terminal device 101 and server 103, such as a wired communication link or a wireless communication link.

[0039] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, the server can be a server group composed of multiple server devices. In addition, the technical solution provided in this application embodiment can be applied to server 103, or to terminal device 101, or it can be implemented by both terminal device 101 and server 103. This application does not impose any special limitations on this.

[0040] In one embodiment of this application, a user performs a business operation through a terminal device 101 and sends a business request to a server 103 via a network 102. The server 103 then processes the business request and returns a processing result. This business operation could specifically include online shopping, participating in a flash sale, or online appointment booking. During the business processing, the business system contains multiple nodes, and business data flows between these nodes. To ensure that the user's submitted business request is processed correctly, it is necessary to ensure that data is not missing, lost, or transmitted incorrectly during transmission. Accordingly, maintenance personnel need to verify the data in a timely manner or when system anomalies are detected, or set a preset data verification service interval to verify the data in the system. During data verification, data between any two adjacent nodes in each business line within the business system can be verified. The node sending data is the source node, and the node receiving data is the destination node. A multi-version concurrency control mechanism is then used to perform consistent reads on the data between the source and destination nodes to obtain a first data copy corresponding to the source node and a second data copy corresponding to the destination node. The data in the first and second data copies is then compared to obtain the data verification result, and the target operation is performed based on the result. The data verification result is categorized into two types: abnormal data transmission and normal data transmission. If the data transmission is normal, no processing is required, and the system awaits the next data verification. If the data transmission is abnormal, the data in the destination node needs to be repaired based on the correct data in the source node.

[0041] In one embodiment of this application, the data verification service for data verification can be built on server 103 or on terminal device 101. When the triggering conditions for data verification are met, the data verification service can be invoked to verify the data of the source and destination nodes to be verified. In this embodiment of the application, the data verification service is an external service. By setting up the data verification service separately, it is possible to avoid affecting other services and data transmission in the system, and to reduce the workload of modifying system code.

[0042] Furthermore, when forming the first and second data copies, they can be generated based on the primary key information and version number in each row of data. The primary key information is the data that can distinguish each row of data, and the version number is used to indicate how many times the data in the data row has been changed. Under normal circumstances, the version number corresponding to a certain primary key information in the source node is the same as the version number corresponding to the same primary key information in the destination node. If there are differences or missing versions, it indicates that an anomaly has occurred during the data transmission process.

[0043] In one embodiment of this application, a cloud server providing cloud computing services can be used to execute the data processing method in this application. Accordingly, the technical solution of this application relates to cloud computing and cloud storage in cloud technology.

[0044] Cloud computing is a computing model that distributes computing tasks across a large pool of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" appear infinitely scalable, readily available, on-demand, and expandable, with payment based on usage.

[0045] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0046] Based on logical function, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS (Infrastructure as a Service) layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is a platform for running software, such as databases and web containers. SaaS refers to various types of business software, such as web portals and bulk SMS senders. Generally speaking, SaaS and PaaS are upper layers compared to IaaS.

[0047] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0048] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0049] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0050] Figure 2 The diagram illustrates a data verification process using locking mechanisms in related technologies, such as... Figure 2 As shown, the flow of business data is Node 1 → Node 2 → Node 3. When the operations and maintenance personnel want to verify the data between Node 2 and Node 3, they first send a data verification request to Node 2. After receiving the data verification request, Node 2 pauses receiving new data from Node 1 and waits for the existing data to be transmitted to Node 3 before returning. The operations and maintenance personnel then obtain data from both Node 2 and Node 3 and begin the verification. Finally, they send any discrepancies to Node 3 to complete the verification.

[0051] Although Figure 2 The proposed solution has little impact on systems with small data volumes. However, for systems with large data volumes, pausing data transmission between two nodes by locking them for data verification will consume a large amount of resources, affect normal data transmission, and thus seriously impact user services.

[0052] In view of the problems existing in the related technologies, the technical solutions such as the data processing method, data processing device, computer-readable medium and electronic device provided in this application are described in detail below with reference to specific embodiments.

[0053] Figure 3The schematic diagram illustrates a flowchart of a data processing method in one embodiment of this application. This data processing method can be executed by a data verification service configured in a server. Specifically, the server may be... Figure 1 The server 103 in the middle can also be executed by the data verification service configured in the terminal device 101. For example Figure 3 As shown, the data processing method in this application embodiment can mainly include the following steps S310 to S320.

[0054] S310: In response to a data verification request, obtain a first data copy corresponding to the source node and a second data copy corresponding to the destination node. The first data copy contains first primary key information and a first version number corresponding to each piece of data in the source node, and the second data copy contains second primary key information and a second version number corresponding to each piece of data in the destination node.

[0055] S320: Compare the first primary key information and the first version number with the second primary key information and the second version number respectively to obtain the data verification result, and perform the target operation based on the data verification result.

[0056] The data processing method provided in this application can, on the one hand, obtain a first data copy corresponding to the source node and a second data copy corresponding to the destination node after responding to a data verification request. By comparing the first primary key information and the first version number in the first data copy with the second primary key information and the second version number in the second data copy, it can be determined whether the data sent by the source node and the data received by the destination node are the same. Since data copies are used, there is no need to stop data transmission or lock the transmitted data, thus realizing lock-free data verification and reducing the occupation of system resources. On the other hand, since this application only forms the first data copy and the second data copy based on the primary key information and version number in each row of data, the amount of data used in data verification is reduced, and the efficiency of data verification is improved.

[0057] The following section provides a detailed explanation of the specific implementation methods for each step of the data processing method.

[0058] In S310, in response to a data verification request, a first data copy corresponding to the source node and a second data copy corresponding to the destination node are obtained. The first data copy contains first primary key information and a first version number corresponding to each piece of data in the source node, and the second data copy contains second primary key information and a second version number corresponding to each piece of data in the destination node.

[0059] In one embodiment of this application, the conditions for triggering a data verification task are mainly divided into two situations: one is that the operation and maintenance personnel discover an anomaly during system operation, triggering the data verification task; the other is that the system actively triggers the data verification task at a predetermined time, such as actively triggering the data verification task during off-peak business hours. Whether triggered by operation and maintenance personnel or actively by the system, after responding to the data verification request, the data verification service can retrieve the data corresponding to the source node and destination node to be verified from the data tables maintained at the source node and destination node according to the time point of the response to the data verification request, in order to perform data verification. The source node and destination node to be verified can be nodes designated by operation and maintenance personnel or the system, or some or all nodes in various business threads within the system.

[0060] In one embodiment of this application, the system can be either a distributed system or a non-distributed system. During data verification, multiple node groups consisting of source nodes and destination nodes can be verified simultaneously. When performing data verification based on the acquired data corresponding to the source and destination nodes to be verified, data verification can be performed using a first data copy corresponding to the source node and a second data copy corresponding to the destination node. The first and second data copies include primary key information and version numbers corresponding to each data entry in the data table. This avoids locking the business threads between the source and destination nodes for data verification, thereby preventing disruption to normal external data transmission and impacting user services.

[0061] Figure 4 A flowchart illustrating the process of obtaining the first and second data copies is shown, as follows: Figure 4 As shown, this process includes at least S401-S403:

[0062] In S401, a snapshot read is performed on the data corresponding to the source node and the data corresponding to the destination node based on the multi-version concurrency control mechanism to obtain a first snapshot corresponding to the source node and a second snapshot corresponding to the destination node.

[0063] In one embodiment of this application, the consistent read mechanism in MVCC, namely snapshot read, is mainly adopted. By performing snapshot reads on the data corresponding to the source node and the data corresponding to the destination node, a first snapshot corresponding to the source node and a second snapshot corresponding to the destination node can be obtained. According to the mechanism of consistent read (snapshot read), the data contained in the first snapshot is the transaction-committed updated data that can be read from the data table corresponding to the source node at the time of responding to the data verification request. Similarly, the data contained in the second snapshot is also the transaction-committed updated data that can be read from the data table corresponding to the destination node at the time of responding to the data verification request.

[0064] In S402, the first primary key information and the first version number corresponding to each piece of data are extracted from the first snapshot, and the second primary key information and the second version number corresponding to each piece of data are extracted from the second snapshot.

[0065] In one embodiment of this application, since each node's corresponding data table contains multiple data entries, and each data entry contains data corresponding to multiple dimensions, in order to reduce the amount of data used during data verification without affecting the data verification results, after obtaining the first snapshot and the second snapshot, data of key dimensions that can be used to indicate data update information can be extracted. In this embodiment of the application, the key dimension data specifically includes the primary key information and version number corresponding to each data entry. The primary key information is used to distinguish each data entry, such as node number, node name, etc. The version number is used to indicate the update status of the data in each data row. It is a globally auto-incrementing version number. When the data in a data row changes, the version number corresponding to that data row can increment by 1 or by other fixed values. This embodiment of the application does not specifically limit this. Since the version number corresponding to a row will be updated when the data in a certain row changes, the location of the system data change can be known based on the value of the version number. For example, if there is a data record with primary key A and version number 3, it means that the data in the data row corresponding to A has been changed 3 times before the time of responding to the data verification request.

[0066] In one embodiment of this application, after obtaining the first snapshot and the second snapshot, the first primary key information and the first version number corresponding to each piece of data can be extracted from the first snapshot, and the second primary key information and the second version number corresponding to each piece of data can be extracted from the second snapshot. Then, a first data copy can be generated based on the first primary key information and the first version number, and a second data copy can be generated based on the second primary key information and the second version number.

[0067] Taking routing data between network controllers as an example, each network controller controls multiple Virtual Routing Forwarding (VRF) operations. Any two adjacent network controllers are designated as the source node and destination node, respectively. According to the method described in the above embodiments, snapshots and data copies corresponding to the source and destination nodes can be obtained. Tables 1 and 2 show the first snapshot corresponding to the source node and the second snapshot corresponding to the destination node:

[0068] Table 1 shows the first snapshot corresponding to the source node.

[0069]

[0070] Table 2 shows the second snapshot corresponding to the destination node.

[0071]

[0072] Analysis of Tables 1 and 2 shows that the data rows in the first and second snapshots contain five dimensions of data: VRF, Cidr (Classless Inter-Domain Routing), A, B, and version number. A and B can be VRF-related information such as the next-hop IP address and prefix. Since a data row can be uniquely identified based on the VRF and Cidr, the VRF, Cidr, and version number can be extracted from the first and second snapshots to form the first and second data copies. Tables 3 and 4 show the first and second data copies.

[0073] Table 3 shows the first data replicas corresponding to the source node.

[0074]

[0075] Table 4 shows the second data copy corresponding to the destination node.

[0076]

[0077] In S403, the first data copy is generated based on the first primary key information and the first version number, and the second data copy is generated based on the second primary key information and the second version number.

[0078] In one embodiment of this application, after obtaining the first primary key information, the first version number, the second primary key information, and the second version number, a first data copy can be generated based on the first primary key information and the first version number, and a second data copy can be generated based on the second primary key information and the second version number. Then, by comparing the data in the first data copy and the second data copy, it can be determined whether there are any anomalies in the data flow from the source node to the destination node.

[0079] In one embodiment of this application, since the first version number and the second version number are globally auto-incrementing, the source node and the destination node should have the same primary key information and the same version number corresponding to the same primary key information, provided that no abnormality occurs during the data transmission process. If the primary key information in the destination node is different from the primary key information in the source node, or if the version numbers corresponding to the same primary key information in the source node and the destination node are different, it indicates that there is an abnormality in the data transmission process. Therefore, in the embodiment of this application, it can be determined whether there is an abnormality when the data flows from the source node to the destination node by comparing whether the primary key information in the source node and the destination node are the same and the version numbers corresponding to the same primary key information in the source node and the destination node.

[0080] In one embodiment of this application, while updating the version number based on changes in the data in the data row, a log can also be generated based on the updated data. This log records the data before and after the update. Both the source node and the destination node have corresponding logs. By comparing the logs of the source node and the destination node, it can be determined whether the data changes are consistent. If they are inconsistent, a rollback can be performed based on the data in the source node's log to repair the data rows in the destination node. This ensures that when the version numbers corresponding to the same primary key information are the same in the source and destination nodes, the data in all data rows with version numbers lower than that version number remains consistent across the source and destination nodes.

[0081] In S320, the first primary key information and the first version number are compared with the second primary key information and the second version number respectively to obtain the data verification result, and the target operation is performed based on the data verification result.

[0082] In one embodiment of this application, after obtaining the first data copy and the second data copy, the first primary key information and the first version number in the first data copy can be compared with the second primary key information and the second version number in the second data copy to obtain the data verification result. In this embodiment, the data verification result is of two types: the first type is normal data transmission, and the second type is abnormal data transmission. When the data transmission is normal, no processing of the system data is required; when the data transmission is abnormal, the erroneous data needs to be repaired based on the correct data. Furthermore, data transmission anomalies can be categorized into two types. The first type is data loss, as shown in Tables 3 and 4, where the second data copy is missing the corresponding data row (VRF2, 172.21.0.3) compared to the first data copy. The second type of data transmission anomaly is data error. For example, if the second data copy shown in Table 4 contains the corresponding data row (VRF2, 172.21.0.3), but the version number of that row is different from the version number of (VRF2, 172.21.0.3) in the first data copy, then an error has occurred during data transmission. Therefore, when judging data transmission anomalies, two methods are needed.

[0083] Figure 5 A flowchart illustrating a process for obtaining data verification results is shown, such as... Figure 5 As shown, in S501, the first primary key information is compared with the second primary key information; in S502, when the first primary key information and the second primary key information are the same, the first target version number corresponding to the first primary key information and the second target version number corresponding to the second primary key information are obtained; in S503, the first target version number is compared with the second target version number; in S504, when the first target version number and the second target version number are different, the data verification result is determined to be a data transmission anomaly.

[0084] In this process, there can be one or more first primary key pieces of information and second primary key pieces of information. When there are multiple first primary key pieces of information and multiple second primary key pieces of information, the first primary key pieces of information are polled, and any one of them is taken as the first target primary key piece of information. Then, the first target primary key piece of information is compared with each of the second primary key pieces of information. When a second target primary key piece of information that is identical to the first target primary key piece of information is found, the first target version number corresponding to the first primary key piece of information and the second target version number corresponding to the second target primary key piece of information are obtained. Then, the first target version number and the second target version number are compared, and the data verification result is determined based on the comparison result. Similarly, the above method can also be used to obtain the data verification result when there is only one first primary key piece of information and / or one second primary key piece of information.

[0085] Figure 5 The method shown can identify data transmission anomalies caused by data transmission errors. Simultaneously, by comparing the first and second primary key information, if no second primary key information exists that matches the first primary key information, it indicates that data loss occurred during the data flow from the source node to the destination node, thus confirming the data verification result as a data transmission anomaly. Taking the first and second data replicas shown in Tables 3 and 4 as examples, the primary key information (VRF1,10.206.0.1), (VRF1,10.206.0.2), (VRF2,172.21.0.1), and (VRF2,172.21.0.3) in the first data replica are compared with the primary key information (VRF1,10.206.0.1), (VRF1,10.206.0.2), and (VRF2,172.21.0.1) in the second data replica. The comparison shows that the primary key information (VRF1,10.206.0.1), (VRF1,10.206.0.2), and (VRF2,172.21.0.3) in the first data replica are more accurate than those in the second data replica. The primary key information (VRF1, 10.206.0.1), (VRF1, 10.206.0.2), and (VRF2, 172.21.0.1) in the first data copy are identical to those in the second data copy. The version numbers corresponding to these primary key information can then be compared. The comparison shows that the version numbers for each primary key information are also identical, indicating that no abnormalities occurred during data transmission. However, the primary key information (VRF2, 172.21.0.3) is missing in the second data copy, indicating that this data was lost during data transmission and needs to be added to the destination node's data table.

[0086] In one embodiment of this application, whether it is data loss or data error, it is a data transmission anomaly, and the data corresponding to the destination node needs to be repaired to ensure that the data received by the destination node is the same as the data sent by the source node, thereby avoiding any impact on the processing results of user services.

[0087] In one embodiment of this application, when performing data repair, it is necessary to lock the business thread corresponding to the data row to be repaired in order to suspend data transmission between the source node and the destination node, and then repair the data table of the destination node according to the correct source node data. Figure 6 The flowchart illustrates the process of performing target operations based on data verification results, such as... Figure 6As shown, in S601, when the data verification result is that the data transmission is abnormal, the target primary key information corresponding to the abnormal data is obtained; in S602, the state of the lock corresponding to the target primary key information is switched to the locked state to stop the data transmission between the source node and the destination node; in S603, the data of the destination node is repaired according to the source node data corresponding to the target primary key information.

[0088] When repairing data on a target node based on source node data corresponding to the target primary key information, the target primary key information can first be compared with the primary key information in the data table corresponding to the source node to obtain the target data corresponding to the target primary key information; then, the target node can be repaired based on the target data.

[0089] Since data transmission anomalies include both data loss and data errors, the methods used for data repair differ. Specifically, when the second data copy contains a version number corresponding to the target primary key information, and this version number differs from the version number in the first data copy (i.e., the data transmission anomaly is a data error), the data to be updated can be retrieved from the data table corresponding to the destination node based on the target primary key information. This updated data is the data corresponding to the target primary key information; then, the updated data is updated based on the retrieved target data. When the second data copy does not contain a second primary key information corresponding to the target primary key information (i.e., the data transmission anomaly is a data loss), the target data can be inserted into the data table corresponding to the destination node to complete the destination node's data table.

[0090] In one embodiment of this application, a distributed lock is used. After determining the target primary key information, the lock's state can be switched to a locked state by assigning a value to the lock corresponding to the target primary key information, facilitating data repair on the target node. A distributed lock is a method for controlling synchronized access to shared resources between distributed systems. If different systems or different hosts within the same system share one or a group of resources, mutual exclusion is often required to prevent interference and ensure consistency when accessing these resources; this necessitates the use of a distributed lock.

[0091] In one embodiment of this application, the distributed lock can be a Redis-based distributed lock or a ZooKeeper-based distributed lock.

[0092] When using a Redis-based distributed lock, a data verification service can be used to record the lock key in Redis. Each source node periodically polls Redis to obtain the key and locks the corresponding business thread based on the key, thereby repairing the data in that business thread. The following explanation of the Redis-based distributed lock will be based on the first and second data replicas shown in Tables 3 and 4.

[0093] Figure 7 A schematic diagram of an interface for a Redis-based distributed lock is shown, such as... Figure 7 As shown, data verification service 701 records a key: lock-VRF2 using Redis 702. Two source nodes 703 and 704 send data to the same destination node 705. When it is determined that the second data replica is missing information corresponding to the primary key information (VRF2, 172.21.0.3) and data repair is required, data verification service 701 sets a value for lock-VRF2 to pause the data transmission of the VRF2 business thread and inserts the data corresponding to (VRF2, 172.21.0.3) from the source node into the data table of the destination node 705.

[0094] In one embodiment of this application, if the data transmission between source nodes 703 and 704 and destination node 705 both involve data transmission of the VRF2 business thread, then source nodes 703 and 704 need to periodically poll Redis to see if the key exists. If it does not exist, the first source node to query, such as source node 703, acquires the lock and suspends the data transmission of the corresponding VRF2 business thread between itself and destination node 705 by locking the key. Source node 704 continues to poll periodically until the lock is released by source node 703. Then, source node 704 acquires the lock and locks it to repair the data on destination node 705. If only the data transmission between source nodes 703 / 704 and destination node 705 involves the VRF2 business thread, then only source nodes 703 / 704 need to lock the key to suspend the data transmission of the corresponding VRF2 business thread between source nodes 703 / 704 and destination node 705.

[0095] Furthermore, after the data is repaired, the distributed lock is deleted, data transmission is restored, and the data cached by the source nodes 703 and 704 from the previous node is sent to the destination node 705.

[0096] In one embodiment of this application, when a key exists in Redis, it means that the lock corresponding to the key is locked and data transmission is not possible. Only when the key does not exist in Redis does it mean that the lock is released and normal data transmission is possible. Therefore, before sending data to the destination node, the source node needs to send a query request to obtain the status of the lock corresponding to the primary key information in Redis based on the primary key information in the query request, and determine whether data can be sent to the destination node based on the lock status.

[0097] In one embodiment of this application, multiple locks corresponding to different services can be set according to different service types. For example, if there are VRF1, VRF2 and VRF3, then three keys can be recorded in Redis: lock-VRF1, lock-VRF2 and lock-VRF3. When it is necessary to repair the data of a node that processes a certain type of service, it is only necessary to set a value for the key corresponding to the service type to pause the corresponding data transmission and perform data repair.

[0098] When the distributed lock is based on ZooKeeper, if multiple source nodes want to pause data transmission between themselves and the destination node for data repair, they can create temporary sequential nodes under a specified node in ZooKeeper. The source node corresponding to the first temporary sequential node acquires the distributed lock first. Other source nodes that have not acquired the distributed lock retrieve the distributed lock deletion event from ZooKeeper. When a lock deletion event exists, it is determined whether it is the first in the temporary sequential nodes. If it is, the distributed lock is acquired; otherwise, the distributed lock deletion event is retrieved.

[0099] By employing a distributed lock to lock the business threads that need to perform data repair between the source and destination nodes, and then unlocking the lock to allow new data transmission after the destination node completes the data repair, normal data transmission for other business threads is ensured, while abnormal data transmission threads are repaired. This guarantees the consistency of business data between the source and destination nodes, thereby avoiding any impact on user business.

[0100] To provide a more intuitive and comprehensive understanding of the data verification process in this application, as well as the process for securely repairing data in case of verification anomalies, a detailed explanation is provided based on the first and second snapshots shown in Tables 1 and 2. Figure 8 The flowchart illustrating the data verification and data repair process is shown, such as... Figure 8As shown, a first data copy 802 can be generated by extracting the first primary key information and the first version number from the first snapshot 801, and a second data copy 804 can be generated by extracting the second primary key information and the second version number from the second snapshot 803. By comparing the first data copy 802 and the second data copy 804, the difference data 805 can be determined, which is the data row corresponding to the primary key information (VRF2, 172.21.0.3). Then, the corresponding data row can be obtained from the first snapshot according to the difference data 805. At the same time, the business thread corresponding to VRF2 is locked by the distributed lock corresponding to VRF2, and the data transmission between the source node and the destination node is suspended. The data table 806 of the destination node is repaired according to the data row corresponding to the difference data 805. Specifically, the data row is inserted into the data table of the destination node.

[0101] In one embodiment of this application, after locking the business thread using a distributed lock and before performing data repair on the destination node, the data in the first and second data copies can be compared again. The data verification results obtained from this comparison are used to confirm the previously obtained data verification results. If the data verification results are consistent and both indicate data transmission anomalies, then data repair is performed on the destination node. This avoids inaccurate data verification results due to large data volumes, thus preventing unnecessary data repair that consumes significant system resources.

[0102] The data processing method in this application obtains a first data copy corresponding to the source node and a second data copy corresponding to the destination node after responding to a data verification request. The first and second data copies contain primary key information and a version number corresponding to each data entry. Then, the first primary key information and first version number in the first data copy are compared with the second primary key information and second version number in the second data copy to obtain the data verification result. Based on the data verification result, the target operation is executed. This data processing method, on the one hand, uses comparisons between data copies corresponding to the source and destination nodes to determine if the data sent by the source node and received by the destination node are the same. Therefore, it does not require stopping external data transmission or locking the data transmission line, achieving lock-free data verification and reducing system resource consumption. On the other hand, since this application only forms the first and second data copies based on the primary key information and version number in each data entry, it reduces the amount of data used during data verification and improves the efficiency of data verification.

[0103] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0104] The following describes an apparatus embodiment of this application, which can be used to execute the data processing method described in the above embodiments of this application. Figure 9 A schematic block diagram of the data processing apparatus provided in an embodiment of this application is shown. Figure 9 As shown, the data processing device 900 includes: a copy acquisition module 910 and a data comparison module 920, specifically:

[0105] The copy acquisition module 910 is used to respond to a data verification request and acquire a first data copy corresponding to the source node and a second data copy corresponding to the destination node. The first data copy contains a first primary key information and a first version number corresponding to each piece of data in the source node, and the second data copy contains a second primary key information and a second version number corresponding to each piece of data in the destination node. The data comparison module 920 is used to compare the first primary key information and the first version number with the second primary key information and the second version number, respectively, to obtain a data verification result, and to perform a target operation based on the data verification result.

[0106] In some embodiments of this application, based on the above technical solutions, the copy acquisition module 910 includes: a snapshot generation unit, used to perform snapshot reading on the data corresponding to the source node and the data corresponding to the destination node based on a multi-version concurrency control mechanism, to obtain a first snapshot corresponding to the source node and a second snapshot corresponding to the destination node; an information extraction unit, used to extract the first primary key information and the first version number corresponding to each piece of data from the first snapshot, and extract the second primary key information and the second version number corresponding to each piece of data from the second snapshot; and a copy generation unit, used to generate the first data copy according to the first primary key information and the first version number, and generate the second data copy according to the second primary key information and the second version number.

[0107] In some embodiments of this application, based on the above technical solutions, the data comparison module 920 is configured to: compare the first primary key information with the second primary key information; when the first primary key information and the second primary key information are the same, obtain the first target version number corresponding to the first primary key information and the second target version number corresponding to the second primary key information; compare the first target version number with the second target version number; when the first target version number and the second target version number are different, determine that the data verification result is a data transmission anomaly.

[0108] In some embodiments of this application, based on the above technical solutions, the data comparison module 920 is configured to: compare the first primary key information with the second primary key information, and when there is no second primary key information that is the same as the first primary key information, determine that the data verification result is a data transmission anomaly.

[0109] In some embodiments of this application, based on the above technical solutions, the data processing device 900 is configured to: before responding to a data verification request, respond to a query request sent by the source node, obtain the lock status corresponding to the primary key information in the query request; when the lock status is unlocked, transmit the data to the destination node through the source node.

[0110] In some embodiments of this application, based on the above technical solutions, the data comparison module 920 includes: an acquisition unit, used to acquire target primary key information corresponding to the abnormal data when the data verification result is an abnormal data transmission; a locking unit, used to switch the state of the lock corresponding to the target primary key information to a locked state to stop data transmission between the source node and the destination node; and a first repair unit, used to repair the data of the destination node according to the source node data corresponding to the target primary key information.

[0111] In some embodiments of this application, based on the above technical solutions, the repair unit includes: a comparison unit, used to compare the target primary key information with the primary key information in the data table corresponding to the source node, so as to obtain target data corresponding to the target primary key information; and a second repair unit, used to repair the target node according to the target data.

[0112] In some embodiments of this application, based on the above technical solutions, when the version number in the second data copy corresponding to the target primary key information is different from the version number in the first data copy corresponding to the target primary key information, the second repair unit is configured to: obtain the data to be updated from the data table corresponding to the target node according to the target primary key information; and update the data to be updated according to the target data.

[0113] In some embodiments of this application, based on the above technical solutions, when there is no second primary key information in the second data copy that is the same as the target primary key information, the second repair unit is configured to insert the target data into the data table corresponding to the destination node.

[0114] In some embodiments of this application, based on the above technical solutions, the data comparison module 920 is configured to: before performing data repair on the target node based on the source node data corresponding to the target primary key information, compare the data in the first data copy and the second data copy again to confirm the data verification result.

[0115] In some embodiments of this application, the source node and the destination node are any two adjacent nodes in a service line containing multiple nodes; based on the above technical solution, the data processing device 900 is further configured to: after updating the data table corresponding to the destination node according to the target data, transmit the cached data to the destination node through the source node, wherein the cached data is the data sent to the source node by the previous node adjacent to the source node.

[0116] In some embodiments of this application, based on the above technical solutions, the lock is a distributed lock based on Redis or a distributed lock based on ZooKeeper.

[0117] The specific details of the data processing apparatus provided in the various embodiments of this application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0118] Figure 10 This schematically illustrates a computer system architecture block diagram for an electronic device used to implement embodiments of the present application. The electronic device may be, for example... Figure 1 The terminal device 101 and server 103 shown are illustrated.

[0119] It should be noted that, Figure 10 The computer system 1000 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0120] like Figure 10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 1002 or programs loaded from storage section 1008 into random access memory (RAM). The RAM 1003 also stores various programs and data required for system operation. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output interface 1005 (I / O interface) is also connected to the bus 1004.

[0121] In some embodiments, the following components are connected to the input / output interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a local area network card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the input / output interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0122] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1009, and / or installed from removable medium 1011. When the computer program is executed by central processing unit 1001, it performs various functions defined in the system of this application.

[0123] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable medium, or any combination of the above. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0125] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0126] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause an electronic device to execute the method according to the embodiments of this application.

[0127] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0128] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, include: In response to the data verification request, a snapshot read is performed on the data corresponding to the source node and the data corresponding to the destination node based on the multi-version concurrency control mechanism to obtain a first snapshot corresponding to the source node and a second snapshot corresponding to the destination node; Extract the first primary key information and the first version number corresponding to each piece of data from the first snapshot, and extract the second primary key information and the second version number corresponding to each piece of data from the second snapshot; A first data copy is generated based on the first primary key information and the first version number, and a second data copy is generated based on the second primary key information and the second version number; The first primary key information and the first version number are compared with the second primary key information and the second version number respectively to obtain the data verification result. The data verification result is used to indicate whether the data transmission between the source node and the destination node is normal.

2. The method according to claim 1, characterized in that, The step of comparing the first primary key information and the first version number with the second primary key information and the second version number respectively to obtain data verification results includes: Compare the first primary key information with the second primary key information; When the first primary key information is the same as the second primary key information, obtain the first target version number corresponding to the first primary key information and the second target version number corresponding to the second primary key information; Compare the first target version number with the second target version number; When the first target version number is different from the second target version number, the data verification result is determined to be a data transmission anomaly.

3. The method according to claim 1, characterized in that, The step of comparing the first primary key information and the first version number with the second primary key information and the second version number respectively to obtain data verification results includes: Compare the first primary key information with the second primary key information; If no second primary key information is found that is identical to the first primary key information, the data verification result is determined to be a data transmission anomaly.

4. The method according to claim 1, characterized in that, Before responding to a data verification request, the method further includes: In response to the query request sent by the source node, obtain the lock status corresponding to the primary key information in the query request; When the lock is in the unlocked state, the data is transmitted to the destination node through the source node.

5. The method according to claim 1, characterized in that, The method further includes: When the data verification result indicates a data transmission anomaly, obtain the target primary key information corresponding to the anomaly data; Switch the state of the lock corresponding to the target primary key information to the locked state to stop data transmission between the source node and the destination node; Data repair is performed on the target node based on the source node data corresponding to the target primary key information.

6. The method according to claim 5, characterized in that, The step of repairing the target node based on the source node data corresponding to the target primary key information includes: The target primary key information is compared with the primary key information in the data table corresponding to the source node to obtain the target data corresponding to the target primary key information; Data repair is performed on the target node based on the target data.

7. The method according to claim 6, characterized in that, When the version number corresponding to the target primary key information in the second data copy is different from the version number corresponding to the target primary key information in the first data copy; The step of repairing the target node based on the target data includes: The data to be updated is obtained from the data table corresponding to the target node based on the target primary key information. The data to be updated is updated based on the target data.

8. The method according to claim 6, characterized in that, When the second data copy does not contain second primary key information that is identical to the target primary key information; The step of repairing the target node based on the target data includes: Insert the target data into the data table corresponding to the destination node.

9. The method according to claim 6, characterized in that, The source node and the destination node are any two adjacent nodes in a business thread that contains multiple nodes. After performing data repair on the target node based on the target data, the method further includes: The cached data is transmitted from the source node to the destination node, wherein the cached data is data sent to the source node by the previous node adjacent to the source node.

10. A data processing apparatus, characterized in that, include: The replica acquisition module is used to respond to data verification requests and perform snapshot reads on the data corresponding to the source node and the data corresponding to the destination node based on a multi-version concurrency control mechanism to obtain a first snapshot corresponding to the source node and a second snapshot corresponding to the destination node; extracts the first primary key information and the first version number corresponding to each piece of data from the first snapshot, and extracts the second primary key information and the second version number corresponding to each piece of data from the second snapshot; A first data copy is generated based on the first primary key information and the first version number, and a second data copy is generated based on the second primary key information and the second version number; The data comparison module is used to compare the first primary key information and the first version number with the second primary key information and the second version number respectively to obtain data verification results. The data verification results are used to indicate whether the data transmission between the source node and the destination node is normal.

11. A computer-readable medium, characterized in that, Includes instructions that, when run on a computer, cause the computer to perform the data processing method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is used to invoke the executable instructions to implement the data processing method according to any one of claims 1 to 9.

13. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer performs the data processing method according to any one of claims 1 to 9.

Images