A metadata-based data exchange method

Abstract

The application provides a metadata-based data exchange method and system, and solves the problem of difficult guarantee and traceability of data transfer security in conventional cross-network and cross-region data exchange. The main scheme comprises the following steps: for a starting node A, a target node C and a plurality of transfer nodes B passing through in data exchange from A to C, a delivery relationship from A to C and a transmission relationship between adjacent nodes are created; delivery layer metadata and a signature are generated by the starting node, and transmission layer metadata and a signature are generated by a transmission sending end; between adjacent nodes, a transmission sending end of a previous node only puts a data file to be transmitted into a transmission cache area and generates corresponding transmission layer metadata, the transmission sending end of the previous node sends the metadata file and other data files to a transmission receiving end of a next node, and the transmission receiving end of the next node performs real-time monitoring; after all the files are received and verified, the target node C unpacks and stores data information.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to a data exchange method based on metadata. Background Technology

[0002] In today's enterprise environment, data security is a critical concern. To protect confidential data, many companies implement internal and external network isolation measures, and even internal networks are often further isolated based on different security levels. This means that data cannot be directly exchanged between internal and external networks and must be exchanged through relay devices. On the other hand, some companies have multiple branches around the world, and network connectivity between these regions may be affected by factors such as geographical distance, political restrictions, or network traffic bottlenecks. These also necessitate data relay devices to improve transmission performance.

[0003] For this type of cross-network, cross-regional data exchange, the originating sender and the final destination lack direct communication. The destination is unaware of which relay devices the data passed through, making it impossible to trace the source of problems. If data is tampered with or lost during transmission, the receiver may not be aware of it. Furthermore, while current data exchange methods can ensure security between devices by encrypting and decrypting data at each relay device, they cannot guarantee against data leakage or tampering at the relay devices themselves. Therefore, a more secure and reliable data exchange method is needed that can verify the integrity and reliability of transmitted data while ensuring data privacy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the existing technology. The present invention proposes a more secure and reliable data exchange method based on metadata.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a data exchange method based on metadata, comprising the following steps:

[0006] S1. For the starting node A, the target node C, and multiple transit nodes B that pass through in the data exchange from A to C, create the delivery relationship from A to C and the transmission relationship between adjacent nodes.

[0007] S2. The starting node A generates delivery layer metadata, encrypts it with the public key of the delivery key, generates a signature with the private key of the sending key, and stores the metadata and signature in the metadata file. Between adjacent nodes, the transmitting end of the previous node puts the data file to be transmitted into the transmission buffer and generates the corresponding transmission layer metadata.

[0008] S3. The transmitting end generates a signature for the metadata using the private key of the transmission key, and stores the metadata and signature together in the metadata file. The transmission buffer of the transmitting end of the previous node sends the metadata file and the corresponding data file to the transmission buffer of the transmitting end of the next node. The transmitting end of the next node monitors the transmission buffer in real time. After discovering a new metadata file, it finds the public key of the corresponding transmission key based on the sending end information in the transmission layer metadata to verify the signature of the metadata.

[0009] S4. After the signature is verified, the target node is determined based on the target node information in the metadata. At the same time, the received data file is verified to be compliant based on the name, size, and hash value in the transport layer metadata file list. If compliant, proceed to the next step. If not, proceed to step S3.

[0010] S5. After all files have been received and verified, the target node C unpacks them and stores the data.

[0011] Furthermore, the specific steps for creating the delivery relationship from A to C in step S1 are as follows:

[0012] The starting node A generates a pair of sending keys S_Key and generates a delivery authorization file based on the sending keys. The delivery authorization file includes the public key of the sending key S_Key and the name and number information of the starting node A.

[0013] After the target node C imports the delivery authorization file, it generates a pair of target keys D_key and generates a delivery confirmation file based on the target keys. The delivery confirmation file includes the public key of the target key D_key and the name, IP and port information of the target node.

[0014] Starting node A imports the delivery confirmation file and creates a delivery relationship.

[0015] Furthermore, the packaging step of the starting node A in step S2 is as follows:

[0016] Divide the large file into n smaller files, and encrypt all the smaller files using a randomly generated key based on AES-256 symmetric encryption to generate an encrypted data file;

[0017] Generate delivery layer metadata Meta_D, which includes the starting node A number and delivery information encrypted with the public key of delivery key D_key;

[0018] A signature is generated on the metadata file by sending the private key of key S_Key, and then stored in the metadata file.

[0019] Furthermore, the delivery information includes sending node information; sender information; target node information; receiver information; file encryption information; file information before encryption; and delivery-related instructions.

[0020] Furthermore, in step S3, transport layer metadata is generated in the first transmission sender, and subsequent transmission senders update the transport layer metadata. The specific steps for subsequent transmission senders to update the transport layer metadata are as follows:

[0021] Add a receiving record to the transmission log. The receiving record includes the receiving node number and the receiving time.

[0022] Add a sending record to the transmission log. The sending record includes the sending node number and the sending time.

[0023] Modify the transmission sender information in the transport layer metadata Meta_T;

[0024] Use the private key of the transport key T_key to generate a signature for the transport layer metadata Meta_T;

[0025] Update the transport layer metadata Meta_T and signature in the metadata file.

[0026] Furthermore, the steps for creating the transmission relationship between adjacent nodes in step S1 are as follows:

[0027] The transmitting end generates a pair of transmission keys T_key, and generates a transmission authorization file based on the transmission keys. The transmission authorization file includes the public key of the transmission key T_key and the name and number information of the transmitting end.

[0028] The receiving end imports the license file and establishes the transmission relationship.

[0029] Furthermore, the target node unpacking step in step S5 is as follows:

[0030] Find the public key of the corresponding sending key S_Key by starting node A number, and verify the signature of the delivery layer metadata Meta_D. If the verification fails, the file is rejected.

[0031] After the signature verification is successful, the delivery information in the delivery layer metadata Meta_D is decrypted using the private key of the target key D_key, and the received file is decrypted using the file encryption information in the delivery information.

[0032] Verify the integrity and unaltered nature of the decrypted files by checking the names, sizes, and hash values ​​of the files in the delivery information, and then merge the split smaller files.

[0033] By using the recipient information in the delivery message, the file and delivery-related instructions are sent to the corresponding recipient on the target node C.

[0034] Retain metadata information for traceability.

[0035] Compared with existing technologies, the metadata-based data exchange method proposed in this invention allows the target node to obtain information about the entire exchange process, including the starting node information, which intermediate nodes it passed through, which files have been delivered, and which files have not yet arrived, as soon as the metadata file arrives at the target node, even if the data file has not been fully transmitted. This improves the traceability of data exchange. During data exchange, the data file and the delivery information in the delivery layer metadata are encrypted throughout the entire process and cannot be tampered with. Neither during transmission nor at intermediate nodes can the actual file content or recipient information be seen or modified. This ensures privacy and security during data transfer while also verifying the integrity and reliability of the transmitted encrypted files based on information such as the number, name, size, and hash value of the files in the transmission layer metadata. This allows for retransmission of erroneous or missing files in case of transmission errors, avoiding the significant time and network resource consumption of discovering errors only after all files have arrived at the target node and needing to start retransmission from the starting node. Attached Figure Description

[0036] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0037] Figure 1 The illustration shows an overall schematic diagram of the delivery process according to an embodiment of the present invention;

[0038] Figure 2 The illustration shows a schematic diagram of the delivery steps according to an embodiment of the present invention;

[0039] Figure 3 The diagram illustrates a transmission process flow according to an embodiment of the present invention. Detailed Implementation

[0040] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0041] According to one embodiment of the present invention, Figures 1-3 As shown.

[0042] This invention provides a data exchange method based on metadata, wherein a device with data sending or receiving capabilities is called a "node".

[0043] This method includes three types of nodes:

[0044] Starting node A: The starting point of the entire data exchange process, and also the data sending end;

[0045] Relay node B: A relay node that passes through in the data exchange from A to C. There may be 0 or more relay nodes, and each node needs to go through two steps: data reception and data transmission.

[0046] Target node C: The node where the data finally arrives, and also the data receiving end.

[0047] like Figure 1 As shown, in this invention, the entire data exchange process from the starting node through the intermediate node to the target node is called delivery, and the data exchange process from one node to the next node is called transmission.

[0048] I. Delivery requires the following steps (see...) Figure 2 ):

[0049] D1 Create delivery relationship

[0050] For multiple deliveries from the same starting node to the same target node, only one delivery relationship needs to be created. The specific steps are as follows:

[0051] Starting node A generates a pair of sending keys S_Key, and generates a delivery authorization file based on the sending keys, which includes, but is not limited to, the public key of the sending keys S_Key and the name and number of the starting node;

[0052] After the target node C imports the delivery authorization file, it generates a pair of target keys D_key. Based on the target keys, it generates a delivery confirmation file, which includes, but is not limited to, the public key of the target key D_key and information such as the name, IP port, etc. of the target node.

[0053] Starting node A imports the delivery confirmation file and creates a delivery relationship.

[0054] The steps for packaging the starting node of D2 are as follows:

[0055] A large file is divided into n smaller files. All files are then encrypted using a randomly generated key based on AES-256 symmetric encryption to generate an encrypted data file.

[0056] Generate delivery layer metadata Meta_D, including but not limited to the following:

[0057] 1. Starting node number;

[0058] 2. Delivery information encrypted with the public key of the delivery key D_key, including but not limited to the following:

[0059] Sending node information (e.g., ID, IP, port, etc.), sender information (e.g., username, etc.); target node information (e.g., ID, IP, port, etc.); receiver information (e.g., username, path, etc.); file encryption information (e.g., encryption key, encryption method, etc.); file information before encryption (e.g., total information: quantity, total size; file list information: file name, path, size, last edit time, hash type, hash value, file name before splitting, etc.); delivery-related instructions (e.g., delivery start time, title, remarks, etc.).

[0060] A signature is generated on the metadata file by sending the private key of key S_Key, and then stored in the metadata file.

[0061]

[0062]

[0063] D3 transmission

[0064] The service that sends data on a node is called the transmission sender, and the service that receives data is called the transmission receiver.

[0065] Before delivery begins, the address of the corresponding target node's receiver is recorded on the transmitting end. During transmission, the next corresponding receiver address is found based on the target node. Accordingly, the transmitting end of the starting node A sends the metadata file and data file to the receiver of the relay node B1, which then sends them to the receiver of B2, and so on, until they reach the receiver of the target node C.

[0066] The specific steps for data processing at the transmitting and receiving ends are described in the following transmission steps description.

[0067] The specific steps for unpacking the target node D4 are as follows:

[0068] Find the public key of the corresponding sending key S_Key by starting node number, and verify the signature of the delivery layer metadata Meta_D.

[0069] If verification fails, the file will be rejected.

[0070] After the signature verification is successful, the delivery information in the delivery layer metadata Meta_D is decrypted using the private key of the target key D_key, and the received file is decrypted using the file encryption information in the delivery information.

[0071] Verify the integrity and unaltered nature of the decrypted files by checking the names, sizes, and hash values ​​of the files in the delivery information, and then merge the split files.

[0072] By using the recipient information in the delivery message, the file and delivery-related instructions are sent to the corresponding recipient on the target node.

[0073] Retain metadata information for easy future traceability.

[0074] II. The transmission requires the following steps (see...) Figure 3 ):

[0075] T1 creates a transport relationship

[0076] For multiple transmissions from the same sender to the receiver, only one transmission relationship needs to be created. The specific steps are as follows:

[0077] The transmitting end generates a pair of transmission keys T_key, and generates a transmission authorization file based on the transmission keys, which includes, but is not limited to, the public key of the transmission key T_key and the name, number, and other information of the transmitting end;

[0078] The receiving end imports the license file and establishes the transmission relationship.

[0079] T2 generates / updates transport layer metadata

[0080] The transmission sender on starting node A generates transport layer metadata. The specific steps are as follows:

[0081] The sending end places the file to be transmitted into the transmission buffer;

[0082] The transmitting end generates transport layer metadata Meta_T for these files, including but not limited to the following:

[0083] Transmitting sender information (e.g., ID, IP, port, interface for requesting resending data, etc.); transmitting receiver information (e.g., ID, IP, port, etc.), file information (e.g., overall information: number of files, total size, etc., list information: file name, path, size, last edit time, hash type, hash value, etc.); transmitting log information (e.g., sending node ID, sending time, etc.); target node information (e.g., ID, IP, port, etc.);

[0084] The transmitting end generates a signature for the transport layer metadata Meta_T using the private key of the transport key T_key, and stores the transport layer metadata Meta_T and the signature together in the metadata file and puts it into the transport buffer.

[0085]

[0086] The transmission sender on relay node B updates the transport layer metadata in the transmission buffer. The specific steps are as follows:

[0087] Add a receiving record (receiving node number, receiving time, etc.) to the transmission log;

[0088] Add a sending record (sending node number, sending time, etc.) to the transmission log;

[0089] Modify the transmission sender information in the transport layer metadata Meta_T;

[0090] Use the private key of the transport key T_key to generate a signature for the transport layer metadata Meta_T;

[0091] Update the transport layer metadata Meta_T and signature in the metadata file.

[0092] T3 Data Transmission: The metadata file is sent from the transmission buffer of the transmitting end to the transmission buffer of the transmitting end, and then other data files to be transmitted are sent. Metadata files and other data files can also be sent simultaneously.

[0093] T4 Data Reception: The receiving end monitors the transmission buffer in real time. Upon discovering a new metadata file, it uses the public key of the corresponding transmission key T_key in the transmission layer metadata Meta_T to verify the signature of the transmission layer metadata Meta_T. If the signatures do not match, the file is rejected. If the signatures match, steps T5 and T6 are initiated simultaneously.

[0094] T5 Target Node Judgment

[0095] Based on the target node information in the transport layer metadata Meta_T, determine whether it is the target node. If it is not the target node, find the next transmission receiver through the target node, proceed to step T2 to update the transport layer metadata, and then send the metadata. After T6 is completed, send the data file.

[0096] If it is the target node, proceed to step D4 after T6 is completed;

[0097] T6 file reception verification:

[0098] The receiving end monitors the transmission buffer in real time. After receiving a new file, it verifies whether the received file matches the file name, size, and hash value in the file list of the transport layer metadata Meta_T in the corresponding metadata file. If it does not match, it requests the sending end to retransmit the file.

[0099] After all files have been received and verified, if the node is not the target node according to the judgment in T5, proceed to step T3 to send data; if it is the target node, proceed to step D4.

[0100] Furthermore, the above-described method steps can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A data exchange method based on metadata, characterized in that, Includes the following steps: S1. For the starting node A, the target node C, and multiple transit nodes B that pass through in the data exchange from A to C, create the delivery relationship from A to C and the transmission relationship between adjacent nodes. S2. The starting node A generates delivery layer metadata, encrypts it with the public key of the delivery key, generates a signature with the private key of the sending key, and stores the metadata and signature in the metadata file. Between adjacent nodes, the transmitting end of the previous node puts the data file to be transmitted into the transmission buffer and generates the corresponding transmission layer metadata. S3. The transmitting end generates a signature for the metadata using the private key of the transmission key, and stores the metadata and signature together in the metadata file. The transmission buffer of the transmitting end of the previous node sends the metadata file and the corresponding data file to the transmission buffer of the transmitting end of the next node. The transmitting end of the next node monitors the transmission buffer in real time. After discovering the new metadata file, it finds the public key of the corresponding transmission key based on the sending end information in the transmission layer metadata to verify the signature of the metadata. S4. After the signature is verified, the target node is determined based on the target node information in the metadata. At the same time, the received data file is verified to be compliant based on the name, size, and hash value in the transport layer metadata file list. If compliant, proceed to the next step. If not, proceed to step S3. S5. After all files have been received and verified, the target node C unpacks them and stores the data.

2. The data exchange method based on metadata according to claim 1, characterized in that: The specific steps for creating the delivery relationship from A to C in step S1 are as follows. The starting node A generates a pair of sending keys S_Key and generates a delivery authorization file based on the sending keys. The delivery authorization file includes the public key of the sending key S_Key and the name and number information of the starting node A. After the target node C imports the delivery authorization file, it generates a pair of target keys D_key and generates a delivery confirmation file based on the target keys. The delivery confirmation file includes the public key of the target key D_key and the name, IP and port information of the target node. Starting node A imports the delivery confirmation file and creates a delivery relationship.

3. The data exchange method based on metadata according to claim 2, characterized in that: The packaging steps for the starting node A in step S2 are as follows: Divide the large file into n smaller files, and encrypt all the smaller files using a randomly generated key based on AES-256 symmetric encryption to generate an encrypted data file; Generate delivery layer metadata Meta_D, which includes the starting node A number and delivery information encrypted with the public key of delivery key D_key; A signature is generated on the metadata file by sending the private key of key S_Key, and then stored in the metadata file.

4. The data exchange method based on metadata according to claim 3, characterized in that: The delivery information includes sending node information; sender information; target node information; receiver information; file encryption information; file information before encryption; and delivery-related instructions.

5. A data exchange method based on metadata according to claim 1, characterized in that: In step S3, transport layer metadata is generated in the first transmission sender, and subsequent transmission senders update the transport layer metadata. The specific steps for subsequent transmission senders to update the transport layer metadata are as follows. Add a receiving record to the transmission log. The receiving record includes the receiving node number and the receiving time. Add a sending record to the transmission log. The sending record includes the sending node number and the sending time. Modify the transmission sender information in the transport layer metadata Meta_T; Use the private key of the transport key T_key to generate a signature for the transport layer metadata Meta_T; Update the transport layer metadata Meta_T and signature in the metadata file.

6. The data exchange method based on metadata according to claim 1, characterized in that: The steps for creating the transmission relationship between adjacent nodes in step S1 are as follows: The transmitting end generates a pair of transmission keys T_key, and generates a transmission authorization file based on the transmission keys. The transmission authorization file includes the public key of the transmission key T_key and the name and number information of the transmitting end. The receiving end imports the license file and establishes the transmission relationship.

7. A data exchange method based on metadata according to claim 1, characterized in that: The specific steps for unpacking the target node in step S5 are as follows: Find the public key of the corresponding sending key S_Key by starting node A number, and verify the signature of the delivery layer metadata Meta_D. If the verification fails, the file is rejected. After the signature verification is successful, the delivery information in the delivery layer metadata Meta_D is decrypted using the private key of the target key D_key, and the received file is decrypted using the file encryption information in the delivery information. Verify the integrity and unaltered nature of the decrypted files by checking the names, sizes, and hash values ​​of the files in the delivery information, and then merge the split smaller files. By using the recipient information in the delivery message, the file and delivery-related instructions are sent to the corresponding recipient on the target node C. Retain metadata information for traceability.

Images