Backtraceable verification method and system based on multi-stage order-preserving alignment

Abstract

The invention belongs to the technical field of data verification, and relates to a backtraceable verification method and system based on multi-stage order-preserving alignment. According to the method, dynamic backtracking verification of the data is realized through addition of the version number attributes and construction of the version number graph, and the problems of dislocation, loss or tampering and the like possibly occurring in the transmission process can be effectively solved. Based on a multi-stage order-preserving alignment mechanism, the problem that verification is difficult due to the fact that versions in multiple areas are inconsistent is solved, and version consistency verification in the whole network range becomes simpler, more convenient and more efficient. According to the invention, through a version number backtracking verification technology, a previous state can be rapidly and accurately traced, and comparison verification is carried out, so that rapid positioning and restoration of system abnormity are realized, and the stability and safety of the system are improved. According to the invention, the integrity and consistency of the data are ensured, and the data loss or damage in the data transmission process is avoided, so that the user experience is improved, and the smoothness and reliability of data transmission are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of data verification, and specifically relates to a traceable verification method and system based on multi-stage order-preserving alignment, which is used to ensure the integrity and accuracy of data transmission or storage. Background Art

[0002] In the process of data transmission and storage, the integrity and accuracy of data are crucial. In order to ensure the correctness of data, various verification technologies are usually used to detect and correct possible errors. However, traditional verification technologies have problems such as low efficiency and irreversible verification results when processing large amounts of data or performing data verification in complex systems.

[0003] Therefore, a new verification technology is needed that can improve the verification efficiency while ensuring the accuracy of data verification and achieve the traceability of the verification process. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a traceable verification method and system based on multi-stage order-preserving alignment. In the traditional data transmission process, data may be misplaced, lost or tampered due to network reasons or other problems, and the multi-stage order-preserving alignment mechanism of the present invention can effectively solve these problems by aligning the data at each stage.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A traceable verification method based on multi-stage order-preserving alignment includes the following steps:

[0007] If the size of the file to be transmitted exceeds the set threshold, the data in the file to be transmitted is divided into blocks to obtain multiple data blocks, each of which has a fixed size;

[0008] The sender stores the version number to be sent into the data block, and then sends the version number and the data block to the receiver;

[0009] The data block transmission process is divided into multiple stages. After receiving the data block, the receiving end of each stage extracts the version number from the data block and uses it for data alignment and backtracking verification.

[0010] Furthermore, the transmitting end stores the version number to be sent in the data block by adding an extra field in the header or tail of the data packet, and the field is used to store the version number of the data block, wherein the version number includes a timestamp and a unique identifier.

[0011] Furthermore, the receiving end uses all data blocks as nodes in the graph, constructs a directed acyclic graph (DAG) based on the previous and next relationships of the data, and forms a version number graph. Specifically, the version number graph is implemented by traversing the data blocks in the data transmission process and connecting them into a directed graph according to the transmission order of the data blocks. Each node in the version number graph represents a data block, and the directed edges between the nodes represent the transmission relationship between the data blocks.

[0012] Furthermore, the transmission process of the data block includes multiple stages, and the receiving end of each stage performs an alignment operation on the received data blocks, and the alignment operation includes calculating the feature vector of each data block, and then using the feature vector to calculate the similarity between the data blocks, and determining the alignment relationship between the data blocks by the similarity. Specifically, the feature vector can be calculated by performing hash calculation on the data block, and then the similarity between the data blocks is quantified using the cosine similarity algorithm, and then the alignment relationship between the data blocks is determined. Among them, the stage refers to the use of a layered transmission design in the data transmission process to ensure the high availability of the system. The receiving end of each layer will perform data alignment operations on the received data. The purpose of this data alignment mechanism is to detect and correct possible data loss, confusion or duplication during the transmission process, so as to ensure that the data can maintain consistency and integrity between different transmission layers. This method can significantly improve the reliability of data transmission and reduce the potential risks caused by network jitter or intermediate node failure. By calculating the feature vector and similarity, a directed graph is constructed to track the data relationship, and whether to merge data blocks or new nodes is determined based on the similarity.

[0013] Furthermore, in the process of data alignment, in order to ensure the accuracy and integrity of data transmission, it is necessary to compare the similarity of each data block. If the similarity of two data blocks exceeds a preset threshold, they are considered to represent different parts of the same data. At this time, the two data blocks will be merged into a new node. The merged node contains the following information:

[0014] 1) Data block content: The integrated data block content will contain all the information in the original data block.

[0015] 2) Unified version number: Extract the version number from the merged data block and mark the new node with a unified version number to indicate that the data blocks belong to the same version.

[0016] This merge operation not only eliminates duplicate and redundant data blocks, but also ensures that subsequent data operations are more efficient and reliable.

[0017] Furthermore, if the similarity between two data blocks does not reach a preset threshold, they are considered to represent different data contents. In this case, each data block will be independently created as a new node, and the original version number in the data block will be directly extracted as the node version information. This can clearly distinguish different data blocks, avoid incorrect merging or data pollution, and provide a clear basis for subsequent data processing.

[0018] Furthermore, in order to realize dynamic backtracking verification of data status, the present invention adopts a version number backtracking verification technology, which includes the following steps:

[0019] During data transmission, each data block is accompanied by a version number attribute, and the version number attributes of all data blocks are recorded in a version number map;

[0020] When the system is abnormal or needs to be backtracked, the version number information in the existing directed acyclic graph (DAG) can be used to dynamically backtrack the data status. This mechanism is based on the structural characteristics of DAG. Through the relationship between nodes, it can quickly locate and trace back to the status of any historical version, ensuring that abnormal situations can be quickly located and handled.

[0021] Furthermore, in the specific implementation process, each node not only stores the content of the data block, but also contains the version number information associated with it and the directed edges pointing to other related nodes. Using this information, dynamic backtracking and verification are performed according to the following steps:

[0022] 1. Locate the target node: According to the abnormal point of the current system or the specified version number, find the corresponding node from the version number map as the starting point of backtracking.

[0023] 2. Path tracing: Using the structural characteristics of directed acyclic graphs, trace back to the specified historical version state layer by layer along the directed edges between nodes. Use algorithms (depth-first search or breadth-first search) to find the shortest path from the current node to the target node.

[0024] 3. Status comparison: During the backtracking process, the system will check the correlation between the current node and the historical node one by one, focusing on verifying the consistency of the version number and the correctness of the data content. Since the content carried by each version number is dynamically updated, there may be differences in the data content between nodes, but it is necessary to ensure that these differences meet system expectations, such as rules or constraints for content changes. At the same time, the version numbers must be consistent and incremental to ensure that there are no version number confusion, jumps or duplications during transmission or storage.

[0025] 4. Exception handling: Based on the results of backtracking and comparison, the system will determine the source, type and scope of the exception. If a version is found to have data inconsistencies during the backtracking process, such as version number jumps, duplications or content verification failures, the system can reload the historical version and restore it to a known correct state by backtracking to load the corresponding historical version. The system will ensure that the recovery operation will not damage the integrity and consistency of other data based on the relationship between version numbers.

[0026] A traceable verification system based on multi-stage order-preserving alignment includes a sending end and a receiving end. If the size of a file to be transmitted exceeds a set threshold, the sending end divides the data in the file to be transmitted into blocks to obtain multiple data blocks, each of which has a fixed size. The sending end then stores the version number to be issued into the data block, and then sends the version number and the data block to the receiving end together. The data block transmission process is divided into multiple stages. After receiving the data block, the receiving end of each stage extracts the version number from the data block and uses it for data alignment and traceback verification.

[0027] The present invention provides a data alignment and verification technology based on the data block version number attribute, which solves the problems of misalignment, loss or tampering that may occur during data transmission, as well as the challenge of verification difficulties due to inconsistent versions in multiple regions. By adding a version number attribute to each data block at the sending and receiving ends of data transmission, and constructing a version number map to perform multi-stage order-preserving alignment operations, dynamic backtracking verification of the data state is achieved. When the system is abnormal, it can quickly trace back to the previous state and perform comparison and verification, effectively dealing with possible problems. In addition, by judging the similarity of data blocks, the integrity and consistency of the data are guaranteed, and the reliability of data transmission is improved. This technical solution provides an efficient and reliable solution for data transmission and has broad application prospects.

[0028] The present invention has the following advantages and beneficial effects:

[0029] 1. Improve data transmission reliability: By adding version number attributes and building a version number map, dynamic backtracking verification of data is achieved, which effectively addresses problems such as misalignment, loss or tampering that may occur during transmission, thereby improving the reliability of data transmission.

[0030] 2. Simplify multi-region version verification: Based on the multi-stage order-preserving alignment mechanism, the problem of inconsistent versions in multiple regions and thus difficult verification is solved, making version consistency verification across the entire network simpler and more efficient. Among them, multiple regions refer to multiple independent regions divided according to geographical locations or computer rooms, so that the system can implement version consistency verification in these regions.

[0031] 3. Rapid exception location and repair: Through version number backtracking verification technology, the previous status can be quickly and accurately traced back and compared and verified, thereby achieving rapid location and repair of system exceptions and improving system stability and security.

[0032] 4. Save network resources: By judging the similarity of data blocks and merging aligned data blocks, unnecessary data redundancy is avoided, the amount of data in the network transmission process is reduced, network resources are saved, and transmission efficiency is improved.

[0033] 5. Improve user experience: Ensure data integrity and consistency, avoid data loss or damage during data transmission, thereby improving user experience and ensuring smooth and reliable data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 .Data block segmentation diagram.

[0035] Figure 2 . Schematic diagram of version number map construction. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below through specific embodiments and drawings.

[0037] In order to make the data transmission process more stable and reliable, the present invention introduces a series of optimization measures based on its own business, including data block segmentation, feature vector calculation, cosine similarity calculation, etc. These optimization measures are aimed at improving the accuracy and efficiency of data alignment, thereby ensuring the integrity and consistency of data transmission.

[0038] Before transmission, the data is divided into blocks, the original text data is preprocessed, and the character encoding is unified using UTF-8 to ensure the format consistency of the text data and facilitate subsequent block operations.

[0039] See attached Figure 1 , the figure shows a schematic diagram of data block segmentation provided by the present invention, the method includes:

[0040] Chunking is performed in a sliding window or fixed-step manner. First, determine the size of the sliding window, i.e., the number of characters in each data block. The window size can be adjusted according to specific requirements and performance considerations. Starting from the beginning of the text data, place the sliding window at the starting position of the text. According to the window size setting, slide the window sequentially and intercept the character sequence of the corresponding size as a data block. The step size of each sliding window is 1 character. During the movement of the sliding window, boundary conditions need to be handled to ensure that no data is lost. When the sliding window moves to the end of the text, the remaining characters need to be processed as the last data block, and each intercepted character sequence is stored as a data block.

[0041] The following are the data chunking algorithm formulas and detailed explanations:

[0042] First, determine the window size W. W is the window size, that is, the number of characters in each data block.

[0043] Initialization: i = 0, B = [], N = the length of the text. The initial index value i is 0, indicating starting from the beginning of the text data. Initialize an empty list B to store the chunked data, and N represents the total length of the text data.

[0044] Traverse the text with a sliding window:

[0045] 1) When i + W < N, perform the following steps:

[0046] Intercept the character sequence as a data block: block = text[i:i + W], where text is the original text data;

[0047] Store the data block block into the list B: apped(block);

[0048] i = i + 1, that is, the sliding window moves one character to the right.

[0049] 2) When i + W ≥ N, it means that the end of the text has been traversed. Perform the following steps:

[0050] Intercept the remaining characters as the last data block: block = text[i:];

[0051] Store the last data block block into the list B: apped(block).

[0052] After chunking and version number recording are completed, construct a version number graph according to the transmission order and version information between data blocks. First, regard each data block as a node and assign a unique identifier to each node. Use the hash value of the data block as the identifier. Then, according to the transmission order between data blocks, construct directed edges to represent the transmission relationship between nodes.

[0053] Construct a directed acyclic graph: traverse each data block in the data transmission process, and connect the nodes into a directed graph according to the transmission order of the data blocks. Figure 2 As shown, for each data block, its adjacent nodes are determined according to its previous and subsequent relationships, and corresponding directed edges are added.

[0054] For each data block B i , there may be a previous data block B i-1 and the next data block B i+1 .

[0055] If B i-1 If it exists, then at node B i and B i-1 Add a directed edge between .

[0056] If B i+1 If it exists, then at node B i and B i+1 Add a directed edge between .

[0057] Let G = (V, E) represent the constructed directed graph, where V represents the node set and E represents the edge set.

[0058] For each data block B i , whose nodes are represented by V i , then:

[0059] If B i-1 If exists, add (V i ,V i-1 ) to the edge set E.

[0060] If B i+1 If exists, add (V i ,V i+1 ) to the edge set E.

[0061] By traversing the data blocks and adding the corresponding directed edges, a directed graph representing the order in which the data blocks are transmitted can be constructed.

[0062] At each stage, the data is aligned, the data is divided into blocks, and the feature vector of each data block is calculated by hashing the data blocks. Then, the cosine similarity algorithm is used to quantify the similarity between the blocks and determine the alignment relationship between the data blocks.

[0063] Cosine similarity algorithm:

[0064]

[0065] Through the above algorithm, the cosine similarity algorithm is used to quantify the similarity between data blocks. In this way, the alignment relationship between data blocks can be determined, and the data alignment operation during data transmission can be realized.

[0066] When an exception occurs in the system and the version number needs to be backtracked, the process is as follows:

[0067] a. According to the description of the abnormal situation, determine the data block where the abnormality occurs.

[0068] b. Starting from the abnormal data block, find the position of the node corresponding to its version number in the version number map, and obtain the neighbor node information (predecessor node and successor node) of the node.

[0069] c. Based on the previous and next relationships of the nodes in the version number graph, use a depth-first search algorithm to traverse the directed graph forward or backward until the nearest node adjacent to the abnormal data block is found, and record the path.

[0070] d. Perform version number comparison and verification on abnormal data blocks and their adjacent nodes to ensure their consistency.

[0071] e. If the verification passes, the abnormal data block is replaced with the corresponding data block in the adjacent node to complete the repair operation; if the verification fails, return to the previous step to continue looking for other possible adjacent nodes.

[0072] f. Repeat the above steps until an effective repair solution is found or all possible paths are traversed.

[0073] Another embodiment of the present invention provides a traceable verification system based on multi-stage order-preserving alignment, including a sending end and a receiving end; if the size of the file to be transmitted exceeds a set threshold, the sending end will block the data in the file to be transmitted to obtain multiple data blocks, each data block has a fixed size, and then the sending end will store the version number to be issued into the data block, and then send the version number together with the data block to the receiving end; the data block transmission process is divided into multiple stages, and the receiving end of each stage extracts the version number from the data block after receiving the data block, and uses it for data alignment and traceback verification.

[0074] The specific working processes of the sending end and the receiving end may refer to the corresponding processes in the aforementioned method embodiment, which will not be repeated here.

[0075] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0076] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0077] The purpose of the above embodiments is to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. A traceable verification method based on multi-stage order-preserving alignment, characterized in that: The following steps are involved: If the size of the file to be transmitted exceeds the set threshold, the data in the file to be transmitted is divided into blocks to obtain multiple data blocks, each of which has a fixed size; The sender stores the version number to be sent into the data block, and then sends the version number and the data block to the receiver; The data block transmission process is divided into multiple stages. After receiving the data block, the receiving end of each stage extracts the version number from the data block and uses it for data alignment and backtracking verification.

2. The method according to claim 1, characterized in that The version number to be sent is stored in the data block by adding an extra field to the header or tail of the data packet, and the field is used to store the version number of the data block; the version number includes a timestamp and a unique identifier.

3. The method according to claim 1, characterized in that The receiving end takes all data blocks as nodes in a graph, constructs a directed acyclic graph according to the previous and next relationships of the data, and forms a version number graph; each node in the version number graph represents a data block, and the directed edges between the nodes represent the transmission relationship between the data blocks.

4. The method according to claim 3, characterized in that The data alignment includes: calculating a feature vector of each data block, calculating similarities between data blocks using the feature vectors, and determining an alignment relationship between data blocks based on the similarities.

5. The method according to claim 4, characterized in that During the data alignment process: If the similarity of two data blocks exceeds a preset threshold, the two data blocks are merged into a new node. The merged new node contains the following information: 1) The content of the data block: The content of the merged data block contains all the information in the original data block; 2) A unified version number: The version number is extracted from the merged data block and a unified version number is marked for the new node; If the similarity between two data blocks does not reach a preset threshold, each data block is independently created as a new node, and the original version number in the data block is directly extracted as the version number of the new node.

6. The method according to claim 4, characterized in that The backtracking verification is to realize dynamic backtracking of data status through the version number map, quickly locate and trace back to the status of any historical version through the previous and next relationships between nodes, and ensure that abnormal situations can be quickly located and handled.

7. The method according to claim 6, characterized in that The backtracking verification comprises the following steps: Locate the target node: According to the current abnormal point or the specified version number, find the corresponding node from the version number map as the starting point of backtracking; Path tracing: Using the structural characteristics of directed acyclic graphs, trace back layer by layer along the directed edges between nodes to the specified historical version state, and use the depth-first search algorithm or breadth-first search algorithm to find the shortest path from the current node to the target node; Status comparison: During the backtracking process, check the correlation between the current node and the historical node one by one, and verify the consistency of the version number and the correctness of the data content; Exception handling: Based on the results of backtracking and comparison, determine the source, type and scope of impact of the exception, and perform exception handling; if data inconsistency is found in a certain version during the backtracking process, load the corresponding historical version through backtracking to restore it to the known correct state; based on the relationship between version numbers, ensure that the recovery operation will not destroy the integrity and consistency of other data.

8. A traceable verification system based on multi-stage order-preserving alignment, characterized in that: It includes a sending end and a receiving end; if the size of the file to be transmitted exceeds a set threshold, the sending end will divide the data in the file to be transmitted into blocks to obtain multiple data blocks, each of which has a fixed size, and then the sending end will store the version number to be issued into the data block, and then send the version number together with the data block to the receiving end; the data block transmission process is divided into multiple stages, and the receiving end of each stage extracts the version number from the data block after receiving the data block, and uses it for data alignment and backtracking verification.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images