Cross-network data exchange method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a cross-network data exchange method, electronic equipment, a storage medium and a program product, and the method comprises the steps: receiving transmission data, determining the identification information of the transmission data according to the service identification, timestamp and segment sequence number of the transmission data, and determining the verification value according to the performance requirement, the security requirement and the data integrity; the hash algorithm is obtained by performing hash operation on the segmented data; according to the identification information, judging whether the transmission data is duplicated data or not; judging whether the transmission data is complete data or not according to the verification value; whether the transmission data is lost data or not is judged according to the segment serial number, if yes, a retransmission instruction is sent to the sending end equipment, the data segments which are repeatedly transmitted can be accurately recognized through the unique identification information, redundant storage and processing are avoided, the data repetition problem is solved from the source, then the integrity of the data is verified according to the verification value, and the data transmission efficiency is improved. And the lost data end is judged according to the segment sequence number, so that the problem of data loss is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to a cross-network data exchange method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] In today's digital era, cross-network data exchange is a key link for enterprises and institutions to realize information sharing and collaborative work. However, the complexity of network environment and the uncertainty in the transmission process make data loss and duplication problems occur frequently, which seriously affects the integrity, accuracy of data and normal operation of business. Therefore, how to improve the security and integrity of cross-network data transmission is a problem that needs to be solved at present. SUMMARY

[0003] Some embodiments of the present application aim to provide a cross-network data exchange method, an electronic device, a storage medium and a program product. Through the technical solutions of the embodiments of the present application, the transmission data sent by the sending end device is received, wherein the transmission data at least includes identification information and a check value of the transmission data, the identification information of the transmission data is determined according to the business identifier, timestamp and segment sequence number of the transmission data, and the check value is obtained by performing hash operation on the segmented data according to the performance requirement, security requirement and data integrity. The transmission data is segmented data. According to the identification information, it is judged whether the transmission data is duplicate data. According to the check value, it is judged whether the transmission data is complete data. According to the segment sequence number, it is judged whether the transmission data is lost data. If it is lost data, a retransmission instruction is sent to the sending end device. In the embodiments of the present application, the identification information, i.e. unique identification information, of the transmission data is generated by combining the business identifier, timestamp and segment sequence number. After the uniqueness verification of Redis or identification storage library, the data segment of repeated transmission can be accurately identified, redundant storage and processing are avoided, the data duplication problem is solved from the source, and then the integrity of the data is checked according to the check value, and the lost data end is also judged according to the segment sequence number. In this way, the problem of data loss can be solved.

[0004] In a first aspect, some embodiments of the present application provide a cross-network data exchange method applied to a receiving end device, wherein the receiving end device and a sending end device are located in different network areas, and the method comprises the following steps: receiving transmission data sent by a sending end device, wherein the transmission data at least includes identification information and a check value of the transmission data, the identification information of the transmission data is determined according to the business identifier, timestamp and segment sequence number of the transmission data, and the check value is obtained by performing hash operation on the segmented data according to the performance requirement, security requirement and data integrity. The transmission data is segmented data. According to the identification information, it is judged whether the transmission data is repeated data. According to the check value, it is judged whether the transmission data is complete data. According to the segment sequence number, it is judged whether the transmission data is missing data, and if it is missing data, a retransmission instruction is sent to the sending end device.

[0005] Some embodiments of the application generate identification information, i.e. unique identification information, of transmission data according to the combination of service identification, timestamp and segment sequence number, which can accurately identify the repeated transmission of data segments after Redis or identification storage library uniqueness verification, avoid redundant storage and processing, solve the data duplication problem from the source, and further check the integrity of the data according to the check value, and judge the missing data end according to the segment sequence number, so that the problem of data loss can be solved.

[0006] Optionally, the transmission data is obtained by splitting the original data by the sending end device using a fixed byte or a logical unit.

[0007] Some embodiments of the application split the original data by the sending end device using a fixed byte or a logical unit to obtain segmented data, i.e. transmission data each time, so that the network occupancy rate of data transmission can be reduced.

[0008] Optionally, the identification information of the transmission data is determined according to the service identification, timestamp and segment sequence number of the transmission data, and includes: The segment sequence number, service identification and timestamp of each segmented data are subjected to hash operation by using a preset hash algorithm, to obtain the identification information of the transmission data, wherein the preset hash algorithm includes at least any one of SHA-256 algorithm, MD5 algorithm, SHA-1 algorithm, SHA-3 algorithm or RIPEMD-160.

[0009] Some embodiments of the application reduce the amount of single transmission data by segmenting according to a fixed byte or a logical unit, generate a globally unique identification, i.e. identification information of transmission data, by using UUID or combined key hash algorithm, and ensure that each data segment is unique in the whole link through the deduplication library index check mechanism.

[0010] Optionally, the check value is obtained by performing hash operation on the segmented data according to performance requirements, security requirements and data integrity, and includes: According to the performance requirements, security requirements and data integrity, different check conditions are determined, and the different check conditions include at least a first check condition and a second check condition; If the segmented data meets the first check condition, a first verification algorithm is used to perform hash operation on the segmented data to obtain the check value. If the segmented data satisfies a second verification condition, a second verification algorithm is used to perform a hash operation on the segmented data to obtain the check value.

[0011] Optionally, the first verification algorithm at least includes an SHA-256 algorithm, and the second verification algorithm at least includes a CRC32 algorithm.

[0012] According to different performance requirements, security requirements, and data integrity, some embodiments of the present application set different verification algorithms to improve verification efficiency.

[0013] Optionally, the determining whether the transmission data is repeated data according to the identification information includes: searching a pre-stored identification repository according to the identification information; If the identification information exists in the pre-stored identification repository, the transmission data is determined to be repeated data.

[0014] Some embodiments of the present application need to perform uniqueness verification after generating a unique identification, i.e., identification information, to ensure that repeated identifications do not occur; the uniqueness of newly generated identification can be checked by querying a local or distributed identification repository; if the identification exists, the identification is regenerated until a unique identification is obtained.

[0015] Optionally, the determining whether the transmission data is complete data according to the check value includes: comparing the check value with a pre-stored check value; If the check value is the same as the pre-stored check value, the transmission data is determined to be complete data.

[0016] Some embodiments of the present application judge the integrity of data in the transmission process by using the detection of the check value, and data loss or errors are found in time.

[0017] Optionally, the determining whether the transmission data is lost data according to the segmented sequence number, and if the transmission data is lost data, sending a retransmission instruction to a sending end device includes: comparing segmented sequence numbers obtained in a preset time period with pre-stored segmented sequence numbers; If the segmented sequence numbers obtained in the preset time period do not match the pre-stored segmented sequence numbers, a lost segmented sequence number is determined; sending a retransmission instruction to a sending end device, wherein the retransmission instruction at least includes the segmented sequence number.

[0018] Some embodiments of the present application receive end equipment detects the segment number fault, can locate the missing segment ID through the transmission log and request retransmission, combined with the maximum retransmission number (M) and the alarm mechanism, to ensure zero loss of data, while providing complete audit trail for troubleshooting.

[0019] Optionally, the method further comprises: According to the timestamp, it is judged whether the transmission data is repeated data.

[0020] Some embodiments of the present application can also judge the timestamp to determine whether the transmission data is repeated data in order to improve the accuracy of verification.

[0021] In a second aspect, some embodiments of the present application provide a cross-network data exchange device applied to a receiving end equipment, wherein the receiving end equipment and a sending end equipment are located in different network areas, and the device comprises: A receiving module is configured to receive transmission data sent by the sending end equipment, wherein the transmission data at least comprises identification information of the transmission data and a check value, the identification information of the transmission data is determined according to a service identification, a timestamp and a segment number of the transmission data, and the check value is obtained by performing a hash operation on the segmented data according to performance requirements, security requirements and data integrity; and the transmission data is segmented data. A first judging module is configured to judge whether the transmission data is repeated data according to the identification information. A second judging module is configured to judge whether the transmission data is complete data according to the check value. A third judging module is configured to judge whether the transmission data is missing data according to the segment number, and if so, send a retransmission instruction to the sending end equipment.

[0022] Some embodiments of the present application can accurately identify the repeated transmission data segment by generating the identification information of the transmission data, i.e. the unique identification information, according to the combination of the service identification, the timestamp and the segment number, and after the Redis or identification repository uniqueness verification, avoid redundant storage and processing, solve the data duplication problem from the source, and further check the data integrity according to the check value, and judge the missing data end according to the segment number, so as to solve the problem of data loss.

[0023] Optionally, the transmission data is obtained by splitting the original data by the sending end equipment using fixed bytes or logical units.

[0024] Some embodiments of the present application split the original data by the sending end equipment using fixed bytes or logical units to obtain the segmented data, i.e. the transmission data each time, so as to reduce the network occupancy rate of data transmission.

[0025] Optionally, the identification information of the transmission data is determined according to a service identifier, a timestamp and a segment sequence number of the transmission data, and includes: The segment sequence number, the service identifier and the timestamp of each segment data are subjected to hash operation by using a preset hash algorithm, to obtain the identification information of the transmission data, wherein the preset hash algorithm at least includes any one of SHA-256 algorithm, MD5 algorithm, SHA-1 algorithm, SHA-3 algorithm or RIPEMD-160.

[0026] Some embodiments of the present application segment by fixed bytes or logical units to reduce the amount of data transmitted at a time, generate a globally unique identification, i.e. the identification information of the transmission data, by using a UUID or a combined key hash algorithm, and ensure that each data segment is unique in the entire link through a deduplication library index verification mechanism.

[0027] Optionally, the check value is obtained by performing hash operation on the segment data according to performance requirements, security requirements and data integrity, and includes: Different check conditions are determined according to the performance requirements, the security requirements and the data integrity, and the different check conditions at least include a first check condition and a second check condition; If the segment data meets the first check condition, a first verification algorithm is used to perform hash operation on the segment data to obtain the check value; If the segment data meets the second check condition, a second verification algorithm is used to perform hash operation on the segment data to obtain the check value.

[0028] Optionally, the first verification algorithm at least includes SHA-256 algorithm, and the second verification algorithm at least includes CRC32 algorithm.

[0029] Some embodiments of the present application set different verification algorithms according to different performance requirements, security requirements and data integrity, to improve verification efficiency.

[0030] Optionally, the first judgment module is configured to: Search in a pre-stored identification storage library according to the identification information; If the identification information exists in the pre-stored identification storage library, determine that the transmission data is duplicate data.

[0031] Some embodiments of the present application need to perform uniqueness verification after generating a unique identification, i.e. the identification information, to ensure that there is no duplicate identification. The uniqueness verification can be performed by querying a local or distributed identification storage library to check whether the newly generated identification already exists. If the identification already exists, the identification is regenerated until a unique identification is obtained.

[0032] Optionally, the second judging module is configured to: compare the check value with a pre-stored check value; if the check value is the same as the pre-stored check value, determine that the transmission data is complete data.

[0033] Some embodiments of the present application determine the completeness of data in the transmission process by detecting the check value, and find data loss or error in time.

[0034] Optionally, the third judging module is configured to: compare the segment sequence number obtained in a preset time period with a pre-stored segment sequence; if the segment sequence number obtained in the preset time period does not match the pre-stored segment sequence, determine the missing segment sequence number; send a retransmission instruction to the sending end device, wherein the retransmission instruction at least includes the segment sequence number.

[0035] When the receiving end device detects a segment sequence number fault, some embodiments of the present application can locate the missing segment ID by transmission log and request retransmission, combine the maximum retransmission number (M) with the alarm mechanism, ensure zero loss of data, and provide a complete audit trail for troubleshooting.

[0036] Optionally, the third judging module is configured to: determine whether the transmission data is repeated data according to the time stamp.

[0037] Some embodiments of the present application can further judge the time stamp to determine whether the transmission data is repeated data, so as to improve the accuracy of verification.

[0038] In a third aspect, some embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cross-network data exchange method of any of the embodiments of the first aspect when executing the program.

[0039] In a fourth aspect, some embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the cross-network data exchange method of any of the embodiments of the first aspect.

[0040] In a fifth aspect, some embodiments of the present application provide a computer program product, including a computer program, wherein the computer program is executable by a processor to implement the cross-network data exchange method of any of the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings needed to be used in some embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0042] Figure 1 A flowchart of a cross-network data exchange method provided by an embodiment of the present application; Figure 2 A flowchart of another cross-network data exchange method provided by an embodiment of the present application; Figure 3 A structural diagram of a cross-network data exchange system provided by an embodiment of the present application; Figure 4 A structural diagram of a cross-network data exchange device provided by an embodiment of the present application; Figure 5 An electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions of some embodiments of the present application will be described below in combination with the drawings in some embodiments of the present application.

[0044] It should be noted that: similar labels and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second" and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0045] In the current digital era, cross-network data exchange is a key link for enterprises and institutions to realize information sharing and collaborative work. However, the complexity of network environment and the uncertainty in the transmission process cause frequent data loss and duplication, which seriously affects the integrity, accuracy of data and the normal operation of business. Therefore, how to improve the security and integrity of cross-network data transmission is a problem that needs to be solved. In view of this, some embodiments of the present application provide a cross-network data exchange method, which comprises receiving transmission data sent by a sending end device, wherein the transmission data at least includes identification information of the transmission data and a check value, the identification information of the transmission data is determined according to the business identification, timestamp and segment sequence number of the transmission data, and the check value is obtained by performing hash operation on the segmented data according to performance requirements, security requirements and data integrity; the transmission data is segmented data; according to the identification information, it is judged whether the transmission data is duplicate data; according to the check value, it is judged whether the transmission data is complete data; according to the segment sequence number, it is judged whether the transmission data is lost data, and if it is lost data, a retransmission instruction is sent to the sending end device. In the embodiments of the present application, the identification information of the transmission data, i.e. the unique identification information, is generated by combining the business identification, timestamp and segment sequence number, which can accurately identify the repeatedly transmitted data segment after the Redis or identification storage library uniqueness verification, avoid redundant storage and processing, solve the data duplication problem from the source, and further check the integrity of the data according to the check value, and judge the lost data end according to the segment sequence number, so as to solve the problem of data loss.

[0046] As shown in Figure 1 The embodiments of the present application provide a cross-network data exchange method, which is applied to a receiving end device, the receiving end device and the sending end device are located in different network areas, and the method comprises: S101, receiving transmission data sent by a sending end device, wherein the transmission data at least includes identification information of the transmission data and a check value, the identification information of the transmission data is determined according to the business identification, timestamp and segment sequence number of the transmission data, and the check value is obtained by performing hash operation on the segmented data according to performance requirements, security requirements and data integrity; the transmission data is segmented data; Specifically, the sending end device and the receiving end device are located in different network areas, the sending end device segments the original data to obtain segmented data, i.e. transmission data each time; the sending end device obtains the business identification, timestamp and segment sequence number of each segment of transmission data, and determines the identification information of the transmission data, i.e. the unique identification information, according to the business identification, timestamp and segment sequence number of the transmission data, and then performs hash operation on the segmented data according to performance requirements, security requirements and data integrity to obtain the check value. The sending end device sends the identification information, check value and segmented data to the receiving end device.

[0047] The unique identifier generated in the embodiment of the application is the core basis for preventing data duplication, and needs to remain unique in the entire data exchange process, regardless of how many nodes and networks the data passes through. After the unique identifier is generated, uniqueness verification needs to be performed to ensure that no duplicate identifier appears. The newly generated identifier can be checked for existence by querying a local or distributed identifier repository. If the identifier exists, the identifier is regenerated until a unique identifier is obtained. The check value calculation is used to detect the integrity of the data in the transmission process and timely detect data loss or errors, which is one of the important means for preventing loss.

[0048] S102, judging whether the transmission data is duplicate data according to the identifier information; Specifically, the receiving end device performs a search in the pre-stored identifier repository according to the obtained identifier information, whether the same identifier information as the identifier information can be found. In this way, it can be judged whether the received transmission data is duplicate transmission data, and duplicate transmission data can be avoided.

[0049] That is, the receiving end device can index query the unique identifier. If duplication occurs, the transmission data is discarded, and the sending end device is returned an acknowledgement information.

[0050] S103, judging whether the transmission data is complete data according to the check value; Further, the receiving end device also compares the check value. If the check values are consistent, the sending end device is returned a response information. If the check values are inconsistent, the sending end device is sent a retransmission instruction.

[0051] S104, judging whether the transmission data is lost data according to the segment sequence number. If it is lost data, the sending end device is sent a retransmission instruction.

[0052] Specifically, the receiving end device also compares the segment sequence number with the pre-stored segment sequence. If the two are different, it indicates that there is a lost segment sequence number. If there is a lost segment sequence number, the sending end device is sent a retransmission instruction, which includes the lost segment sequence number. The sending end device and the receiving end device are coordinated to complete data sending, acknowledgement, duplicate check and integrity check in different network areas. The receiving end device stores the received segment data and tracks the data transmission state. If lost data is detected and triggers retransmission, the duplicate data processing is optimized.

[0053] The sending end device buffers the transmission state of each segmented data, marks the data that has been sent as "to be sent", and sends it in sequence according to the sequence number, starts a timeout timer, initiates retransmission if no acknowledgement information is received, and triggers an alarm if the maximum number of retransmissions is exceeded; at the same time, the sending end device defines "to be sent / sent / received / stored / repeated" states, controls the flow through the state machine mode, and records logs to the transmission log table.

[0054] The receiving end device creates a storage table with a unique identifier as an index, detects unique constraint conflicts when inserting, and skips storage if there is a conflict.

[0055] The embodiments of the present application propose a cross-network data exchange method and system based on anti-loss and anti-repeated information, which realizes zero loss, zero repetition and full-process traceability of data in cross-network transmission through core technologies such as data segmentation, unique identifier generation, bidirectional verification (comparison of check values and repeated detection), state closed-loop management (transmission log and state machine mode), and effectively solves the defects of traditional solutions in data integrity, accuracy and abnormal processing capability.

[0056] Some embodiments of the present application generate identification information, i.e. unique identification information, of transmission data according to the combination of business identifier, timestamp and segment sequence number, which can accurately identify repeated transmission data segments after unique verification by Redis database or identifier storage, avoid redundant storage and processing, solve the data repetition problem from the source, and further check the integrity of the data according to the check value, and judge the missing data end according to the segment sequence number, so as to solve the problem of data loss.

[0057] Another embodiment of the present application further supplements the cross-network data exchange method provided by the above-mentioned embodiments.

[0058] Optionally, the transmission data is obtained by splitting the original data by the sending end device using fixed bytes or logical units.

[0059] The sending end device segments the original data according to fixed bytes or logical units, and adds a unique identifier, a segment sequence number and a check value in each segmented data to obtain the transmission data.

[0060] The fixed bytes can be 50Mb or 100Mb, which are not limited in the embodiments of the present application. The logical unit can be a node corresponding to a business, such as a bill.

[0061] Some embodiments of the present application split the original data by the sending end device using fixed bytes or logical units to obtain segmented data, i.e. transmission data each time, which can reduce the network occupancy rate of data transmission.

[0062] Optionally, the identification information of the transmission data is determined according to a service identifier, a timestamp and a segment sequence number of the transmission data, and includes: The segment sequence number, the service identifier and the timestamp of each segment data are subjected to hash operation by using a preset hash algorithm to obtain the identification information of the transmission data, wherein the preset hash algorithm at least includes any one of SHA-256 algorithm, MD5 algorithm, SHA-1 algorithm, SHA-3 algorithm or RIPEMD-160.

[0063] In the embodiment of the application, a combination key hash algorithm is used to generate a unique identifier, and the segment sequence number , the service primary key (service identifier) , the timestamp are combined to generate a unique identifier ; The formula is: , wherein The SHA-256 algorithm can be used; PK is the service primary key, T is the timestamp, and i is the segment sequence number, and the uniqueness is verified through an identifier storage library; For example, for an order data, the service primary key is , the timestamp is , and the segment sequence number is 3, then ; This method combines the service-related primary key information, time information and segment sequence information, greatly reducing the possibility of duplication; In the embodiment of the application, after the unique identifier is generated, uniqueness verification is required to ensure that there is no duplicate identifier; the local or distributed identifier storage library can be queried to check whether the newly generated identifier already exists; if it exists, the identifier is regenerated until a unique identifier is obtained.

[0064] Some embodiments of the application reduce the amount of data transmitted at a time by segmenting in fixed bytes or logical units, generate a globally unique identifier, i.e., the identification information of the transmission data, by using a UUID or a combination key hash algorithm, and ensure that each data segment is unique in the entire link through a deduplication library index verification mechanism.

[0065] Optionally, the check value is obtained by performing hash operation on the segment data according to performance requirements, security requirements and data integrity, and includes: Different check conditions are determined according to performance requirements, security requirements and data integrity, and the different check conditions at least include a first check condition and a second check condition; If the segment data satisfies the first check condition, a first verification algorithm is used to perform hash operation on the segment data to obtain the check value; If the segmented data satisfies the second check condition, a second verification algorithm is used to perform a hash operation on the segmented data to obtain a check value.

[0066] Optionally, the first verification algorithm at least includes an SHA-256 algorithm, and the second verification algorithm at least includes a CRC32 algorithm (a 32-bit algorithm of cyclic redundancy check).

[0067] Specifically, in the embodiments of the present application, for a scenario with higher requirements for data integrity and less impact on performance, a secure hash algorithm such as SHA-256 is selected; and for a scenario with higher requirements for performance and relatively lower requirements for security, a CRC32 algorithm is a good choice. In the embodiments of the present application, the CRC32 algorithm is taken as an example to calculate a check value for each segmented data. The formula is as follows: In the data transmission process, if any modification (such as encryption, compression, etc.) occurs to the segmented data, the check value needs to be recalculated to ensure that the check value can accurately reflect the current state of the data; at the same time, the check value should be transmitted together with the segmented data to enable the receiving end to perform integrity verification; the check value calculation is used to detect the integrity of the data in the transmission process and discover data loss or errors in a timely manner.

[0068] In some embodiments of the present application, different verification algorithms are set according to different performance requirements, security requirements, and data integrity to improve verification efficiency.

[0069] Optionally, according to the identification information, it is determined whether the transmission data is repeated data, including: According to the identification information, a search is performed in a pre-stored identification storage library; If the identification information exists in the pre-stored identification storage library, it is determined that the transmission data is repeated data.

[0070] Some embodiments of the present application need to perform uniqueness verification after generating a unique identification, i.e., identification information, to ensure that repeated identifications do not occur; the local or distributed identification storage library can be queried to check whether the newly generated identification already exists; if it exists, the identification is regenerated until a unique identification is obtained.

[0071] Optionally, according to the check value, it is determined whether the transmission data is complete data, including: The check value is compared with a pre-stored check value; If the check value is the same as the pre-stored check value, it is determined that the transmission data is complete data.

[0072] ​​​​Some embodiments of the present application determine the integrity of data in the transmission process by employing detection of the check value, and timely find data loss or error.

[0073] Optionally, according to the segment sequence number, it is determined whether the transmission data is lost data, and if so, a retransmission instruction is sent to the sending end device, including: Comparing the segment sequence number obtained in the preset time period with the pre-stored segment sequence; If the segment sequence number obtained in the preset time period does not match the pre-stored segment sequence, the lost segment sequence number is determined; Sending a retransmission instruction to the sending end device, wherein the retransmission instruction at least includes the segment sequence number.

[0074] Specifically, the receiving end detects lost data through sequence number continuity, obtains the missing segment ID and requests retransmission; the sending end queries the cache or historical record to retransmit data, and the receiving end rechecks; content similarity check is added for misjudged duplicate data, and the deduplication library and log table historical data are periodically cleaned up; The receiving end device periodically checks whether the sequence number of the received data segment is continuous with the expected sequence number; it can be realized by maintaining a receiving sequence number list or using a sequence function to identify the storage library; if there is a sequence number fault, it means that a data segment is lost; for example, the expected segment sequence numbers to be received are 1, 2, 3, 4, and 5, and only 1, 2, 4, and 5 are actually received, then it can be judged that the segment with sequence number 3 is lost; at this time, the receiving end obtains the missing segment ID through the transmission log and requests retransmission from the sending end; The sending end queries the local cache or historical record after receiving the retransmission request from the receiving end, finds the corresponding missing data segment and its related information (such as ID_i, CV_i), and re-sends it to the receiving end; the receiving end performs repeated check and integrity check according to the normal receiving process after receiving the retransmitted data segment, and stores it after the check is passed, and updates the state. Some embodiments of the present application receiving end device detect the segment sequence number fault, can locate the missing segment ID through the transmission log and request retransmission, combined with the maximum retransmission number (M) and the alarm mechanism, to ensure zero loss of data, and provide complete audit trail for troubleshooting.

[0075] Optionally, the method further comprises:

[0076] According to the timestamp, it is determined whether the transmission data is duplicate data.

[0077] ​In some cases, duplicate data may be misjudged, such as when the transmission order of data segments is disordered due to network latency, causing the receiving device to mistakenly identify it as duplicate data. To handle this situation, additional verification mechanisms can be added to the receiving device, such as comparing the content similarity of data segments and checking timestamps. If it is found that it may be a misjudgment, the data segment is reprocessed instead of being discarded directly.

[0078] In order to improve the accuracy of verification, some embodiments of this application may also determine the timestamp to determine whether the transmitted data is duplicate data.

[0079] Over time, the data in the deduplication database and transmission log table will continue to increase, affecting system performance. Therefore, it is necessary to clean up historical data regularly. Different retention periods can be set according to the importance and frequency of use of the data. For example, data records that have been successfully stored for more than 3 months and have no abnormalities can be deleted, but a backup should be made before deletion for subsequent auditing and querying.

[0080] like Figure 2 As shown, cross-network data exchange methods include: Step 1: Data preprocessing stage; Step 2, Transmission Process Control Stage; Step 3: Storage and State Management Phase; Step 4: Exception handling and data completion stage; Step 5: Performance optimization and security enhancement.

[0081] like Figure 3 As shown, a cross-network data exchange system based on anti-loss and anti-duplicate information is described. The system includes the following components: Data Acquisition Layer: Used to collect data to be exchanged from the identifier repository, file system, and API interface, and to perform format standardization processing; Data preprocessing layer: Segmentation module: Based on factors such as data size and network bandwidth, the collected data is segmented. It supports splitting by fixed bytes or logical units and adds metadata to each segment, including a unique identifier, total number of segments, current segment number, and checksum. Unique Identifier Generation Module: This module uses a composite key hash algorithm to generate a globally unique identifier for each data segment. The formula is as follows: in, For business primary key, For timestamps, The segment number is used, and the uniqueness of the identifier is verified through a distributed cache or an identifier repository index table. Checksum calculation module: based on or The algorithm calculates a checksum for each data segment for integrity verification during transmission, and automatically recalculates it after data encryption / compression. Transmission Control Layer: Sending terminal module: Utilizes Redis to cache data segments to be sent and their metadata, marks the "to be sent" status, sends data sequentially through multi-threaded / asynchronous tasks, starts a dynamic timeout timer, and supports a retransmission mechanism with a maximum number of retransmissions; The receiving terminal module detects duplicate data through a deduplication database. If the checksums match, it returns an ACK to confirm and stores the data; otherwise, it returns a NACK to request retransmission. Storage management layer: Storage deduplication module: Creates a data table in the target identifier repository with a unique identifier as the unique index. When inserting data, it detects index conflicts. If a conflict occurs, the data is determined to be duplicate data and is skipped from storage. The state management module defines the states as "pending to send / sent / received / stored / repeated / abnormal", controls the state transition through the state machine pattern, and records a complete transmission log for auditing and traceability.

[0082] For example, embodiments of this application can be used for cross-border transaction data synchronization in the financial industry: (1) Scenario requirements The bank headquarters and overseas branches need to synchronize transaction data (such as cross-border remittances and account information) in real time. The data must be transmitted across the Internet / dedicated line with zero loss and zero duplication, while meeting the regulatory requirements for traceability of transaction records.

[0083] (2) Implementation process, such as Figure 2 As shown: (2.1) Data preprocessing Cross-border transaction data is segmented into logical units: for example, each remittance slip is a segment containing fields such as remitter information, recipient information, and amount.

[0084] Generate a unique identifier: Generate a globally unique ID using a composite key hash algorithm (business primary key PK = remittance order number + timestamp T + segment sequence number i), such as ID = HASH("20250526-REM-001" + "20250526143000" + 1).

[0085] Calculate the checksum: Use the SHA-256 algorithm to generate a checksum for each data segment to ensure data integrity.

[0086] (2.2) Transmission process control Sending end (headquarters): Segmented data is cached in Redis, marked as "pending transmission," and sent sequentially to overseas branches via dedicated line. If no ACK confirmation is received within 5 seconds, automatic retransmission is initiated, with a maximum of 3 retransmissions followed by an alarm.

[0087] Receiving end (branch office): After receiving the data segment, first check if the same ID exists in the deduplication database. If it is duplicated, discard it. If the checksum is consistent, return ACK; otherwise, request retransmission.

[0088] (2.3) Storage and State Management Create a transaction record table indexed by a unique ID, and check for unique constraint conflicts during insertion to avoid storing the same remittance data repeatedly.

[0089] The state machine tracks data status: for example, "to be sent" → "sent" (data sent) → "received" (ACK received) → "stored" (written to the local identifier repository). Each state change is recorded in the log table for regulatory auditing.

[0090] (2.4) Exception handling If the receiving end detects that the remittance segment sequence number is not consecutive (such as missing the 3rd segment), it obtains the missing segment ID from the log and requests retransmission to ensure the integrity of the transaction data.

[0091] Regularly clean up deduplication records and logs older than 6 months to free up storage resources.

[0092] (3) Value Zero data loss: Through timeout retransmission and sequence number detection mechanisms, we ensure complete synchronization of cross-border transaction data and avoid the loss of transaction records due to network fluctuations.

[0093] Zero duplicates: Unique IDs and a storage deduplication mechanism prevent the same remittance from being processed repeatedly, avoiding accidental fund operations (such as duplicate deductions).

[0094] Compliance: The entire process is logged to meet anti-money laundering regulatory requirements, and the transmission path and processing status of transaction data can be quickly traced.

[0095] For example, embodiments of this application also include cross-hospital patient data exchange in the English-speaking healthcare industry, specifically including: (1) Scenario requirements The main hospital and its branches need to share sensitive data such as patients' electronic medical records and test reports. Data privacy must be protected when transmitting data across networks, while ensuring that medical records are accurate (key information such as surgical records and medication lists must not be lost or duplicated).

[0096] (2) Implementation process (2.1) Data preprocessing Segmentation by fixed bytes: Large medical image files (such as CT scans) are split into 50MB segments, and text-based medical records are segmented by logical chapters (such as "chief complaint" and "examination results").

[0097] Unique identifier generation: Combine patient ID (PK), timestamp and segment number to generate ID, such as ID=HASH("PAT-2025001"+"20250526150000"+2), and verify uniqueness through distributed cache.

[0098] Checksum calculation: Use CRC32 algorithm for medical record text (performance priority) and SHA-256 for image data (security priority).

[0099] (2.2) Transmission process control Sending end (headquarters): Use multi-threaded asynchronous data segment sending, mark "to be sent" status, and dynamically adjust the timeout time (e.g., set to 10 seconds when the network is unstable).

[0100] Receiving end (branch): Builds a deduplication database using Elasticsearch to quickly query duplicate IDs; returns NACK when the checksum is inconsistent, triggering retransmission.

[0101] (2.3) Storage and State Management The target identifier repository uses a combined index of patient ID and segment ID to prevent duplicate storage of the same segment data for the same patient.

[0102] A new "Abnormal" state has been added to the state machine: If the checksum changes after the data segment is encrypted, the checksum will be automatically recalculated and the state will be marked as "Abnormal", and the administrator will be notified to conduct manual verification.

[0103] (2.4) Exception handling For suspected duplicate medical record data (such as different segments containing similar symptom descriptions), content similarity verification (such as cosine similarity algorithm) is used to determine whether it is a false positive, so as to avoid discarding valid data.

[0104] Regularly clean up logs older than one year, but retain critical operation records related to patient privacy (such as data access and reasons for transmission failures).

[0105] (3) Value Data security: Segmented transmission reduces the risk of single-time leakage, and checksums prevent tampering to ensure the credibility of medical record content.

[0106] Precise synchronization: Unique identification and deduplication mechanisms prevent multiple duplicate medical records of the same patient from existing simultaneously, improving the accuracy of clinical diagnosis.

[0107] Privacy protection: Combining the hospital's existing encryption system (such as encryption of medical record fields) with the verification mechanism in the embodiments of this application forms a dual guarantee.

[0108] In this embodiment, a segmentation strategy and unique identification system during the data preprocessing stage enable refined management of cross-network data and interception of duplicate data. Specifically, segmenting data into fixed bytes or logical units reduces the amount of data transmitted in a single transmission. A globally unique identifier is generated using UUID or a composite key hash algorithm, and a deduplication database index verification mechanism ensures that each data segment is unique throughout the entire transmission chain. For example, order data generates an ID by combining the business primary key, timestamp, and segment sequence number. After uniqueness verification by Redis or an identifier repository, duplicate data segments can be accurately identified and discarded directly, avoiding redundant storage and processing and solving the data duplication problem at its source. Through a two-way verification mechanism and closed-loop status management between the transmission control layer and the storage management layer, data transmission integrity and anomaly traceability are ensured. The sending end caches data segments and marks their status, while the receiving end verifies data integrity through CRC32 / SHA-256 checksum comparison (CV). If the verification fails, a retransmission (NACK) is triggered. Simultaneously, the storage layer uses IDs as unique indexes to deduplicate data, and a state machine model tracks the entire data flow from "pending transmission" to "stored," logging every status change. For example, if the receiving end detects a sequence number gap, it can locate the missing segment ID through the transmission log and request a retransmission. Combined with the maximum number of retransmissions (M) and an alarm mechanism, zero data loss is ensured, while providing a complete audit trail for fault diagnosis.

[0109] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0110] Another embodiment of this application provides a cross-network data exchange apparatus for executing the cross-network data exchange method provided in the above embodiments.

[0111] like Figure 4 The diagram shown is a structural schematic of a cross-network data exchange device provided in an embodiment of this application. It is applied to a receiving device, where the receiving and sending devices are located in different network areas. The cross-network data exchange device includes a receiving module 401, a first judgment module 402, a second judgment module 403, and a third judgment module 404, wherein: The receiving module 401 is used to receive transmission data sent by the sending device. The transmission data includes at least the identification information and the check value of the transmission data. The identification information of the transmission data is determined according to the service identifier, timestamp and segment sequence number of the transmission data. The check value is obtained by performing a hash operation on the segmented data according to performance requirements, security requirements and data integrity. The transmission data is segmented data. The first judgment module 402 is used to determine whether the transmitted data is duplicate data based on the identification information; The second judgment module 403 is used to determine whether the transmitted data is complete based on the check value; The third judgment module 404 is used to determine whether the transmitted data is lost based on the segment sequence number. If the data is lost, a retransmission command is sent to the sending device.

[0112] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0113] Some embodiments of this application generate unique identifier information for transmitted data by combining business identifier, timestamp, and segment sequence number. After uniqueness verification by Redis or an identifier repository, duplicate data segments can be accurately identified, avoiding redundant storage and processing, and solving the data duplication problem from the source. Furthermore, the integrity of the data is verified based on the check value, and the lost data segment is judged based on the segment sequence number. In this way, the problem of data loss can be solved.

[0114] Another embodiment of this application further illustrates the cross-network data exchange device provided in the above embodiments.

[0115] Optionally, the transmitted data is obtained by splitting the original data using fixed bytes or logical units by the sending device.

[0116] Some embodiments of this application use fixed bytes or logical units in the sending device to split the original data into segmented data, i.e., each transmission data, which can reduce the network occupancy rate of data transmission.

[0117] Optionally, the identification information of the transmitted data is determined based on the service identifier, timestamp, and segment sequence number of the transmitted data, including: A preset hash algorithm is used to perform hash operations on the segment number, service identifier, and timestamp of each segment of data to obtain the identification information of the transmitted data. The preset hash algorithm includes at least one of the following: SHA-256 algorithm, MD5 algorithm, SHA-1 algorithm, SHA-3 algorithm, or RIPEMD-160.

[0118] Some embodiments of this application reduce the amount of data transmitted in a single transmission by segmenting data into fixed bytes or logical units, generate a globally unique identifier (i.e., the identifier information of the transmitted data) by combining UUID or composite key hash algorithm, and ensure that each data segment is unique throughout the entire link by using a deduplication database index verification mechanism.

[0119] Optionally, the checksum is obtained by performing a hash operation on the segmented data based on performance requirements, security requirements, and data integrity, including: Based on performance requirements, security requirements, and data integrity, different verification conditions are determined. Each verification condition includes at least a first verification condition and a second verification condition. If the segmented data meets the first verification condition, the first verification algorithm is used to perform a hash operation on the segmented data to obtain the verification value; If the segmented data meets the second verification condition, the second verification algorithm is used to perform a hash operation on the segmented data to obtain the verification value.

[0120] Optionally, the first verification algorithm includes at least the SHA-256 algorithm, and the second verification algorithm includes at least the CRC32 algorithm.

[0121] Some embodiments of this application set different verification algorithms according to different performance requirements, security requirements and data integrity requirements, so as to improve verification efficiency.

[0122] Optionally, the first judgment module is used for: Based on the identification information, a search is performed in a pre-stored identification repository; If the identification information exists in the pre-stored identification repository, the transmitted data is determined to be duplicate data.

[0123] In some embodiments of this application, after generating a unique identifier, i.e., identifier information, a uniqueness verification is required to ensure that no duplicate identifiers appear. This can be done by querying a local or distributed identifier repository to check whether the newly generated identifier already exists. If it does, the identifier is regenerated until a unique identifier is obtained.

[0124] Optionally, the second judgment module is used for: Compare the check value with the pre-stored check value; If the checksum is the same as the pre-stored checksum, the transmitted data is determined to be complete.

[0125] Some embodiments of this application use checksum detection to determine the integrity of data during transmission and promptly detect data loss or errors.

[0126] Optionally, the third judgment module is used for: Compare the segment numbers obtained within the preset time period with the pre-stored segment sequences; If the segment sequence number obtained within the preset time period does not match the pre-stored segment sequence, the lost segment sequence number is determined. Send a retransmission command to the sending device, wherein the retransmission command includes at least the segment sequence number.

[0127] In some embodiments of this application, when the receiving device detects a segment sequence number break, it can locate the missing segment ID through the transmission log and request retransmission. Combined with the maximum number of retransmissions (M) and the alarm mechanism, it ensures zero data loss and provides a complete audit trail for fault diagnosis.

[0128] Optionally, the third judgment module is used for: Determine whether the transmitted data is duplicate data based on the timestamp.

[0129] In order to improve the accuracy of verification, some embodiments of this application may also determine the timestamp to determine whether the transmitted data is duplicate data.

[0130] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0131] It should be noted that each of the implementable methods in this embodiment can be implemented individually or in any combination without conflict. This application does not limit this.

[0132] This application also provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it can implement the operation of any of the methods corresponding to the cross-network data exchange methods provided in the above embodiments.

[0133] This application also provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the cross-network data exchange methods provided in the above embodiments.

[0134] like Figure 5 As shown, some embodiments of this application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement any of the methods included in the above-described cross-network data exchange method.

[0135] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0136] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0137] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0139] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A cross-network data exchange method, characterized in that, Applied to a receiving device, wherein the receiving device and the sending device are located in different network areas, the method includes: The device receives transmission data sent by a sending device, wherein the transmission data includes at least identification information and a checksum. The identification information is determined based on the service identifier, timestamp, and segment sequence number of the transmission data. The checksum is obtained by performing a hash operation on the segmented data based on performance requirements, security requirements, and data integrity. The transmission data is segmented data. Based on the identification information, determine whether the transmitted data is duplicate data; Based on the verification value, determine whether the transmitted data is complete; Based on the segment number, determine whether the transmitted data is lost. If it is lost, send a retransmission command to the sending device.

2. The cross-network data exchange method according to claim 1, characterized in that, The transmitted data is obtained by splitting the original data using fixed bytes or logical units by the sending device.

3. The cross-network data exchange method according to claim 2, characterized in that, The identification information of the transmitted data is determined based on the service identifier, timestamp, and segment sequence number of the transmitted data, including: A preset hash algorithm is used to perform hash operations on the segment number, service identifier, and timestamp of each segment of data to obtain the identification information of the transmitted data. The preset hash algorithm includes at least one of the following: SHA-256 algorithm, MD5 algorithm, SHA-1 algorithm, SHA-3 algorithm, or RIPEMD-160.

4. The cross-network data exchange method according to claim 1, characterized in that, The verification value is obtained by performing a hash operation on the segmented data based on performance requirements, security requirements, and data integrity, including: Based on the performance requirements, security requirements, and data integrity, different verification conditions are determined, and the different verification conditions include at least a first verification condition and a second verification condition. If the segmented data meets the first verification condition, then the first verification algorithm is used to perform a hash operation on the segmented data to obtain the verification value; If the segmented data meets the second verification condition, then the second verification algorithm is used to perform a hash operation on the segmented data to obtain the verification value.

5. The cross-network data exchange method according to claim 4, characterized in that, The first verification algorithm includes at least the SHA-256 algorithm, and the second verification algorithm includes at least the CRC32 algorithm.

6. The cross-network data exchange method according to claim 1, characterized in that, The step of determining whether the transmitted data is duplicate data based on the identification information includes: Based on the identification information, a search is performed in a pre-stored identification repository; If the identification information exists in the pre-stored identification repository, the transmitted data is determined to be duplicate data.

7. The cross-network data exchange method according to claim 1, characterized in that, The step of determining whether the transmitted data is complete based on the check value includes: Compare the verification value with the pre-stored verification value; If the verification value is the same as the pre-stored verification value, the transmitted data is determined to be complete.

8. The cross-network data exchange method according to claim 1, characterized in that, The step of determining whether the transmitted data is lost based on the segment sequence number, and if it is lost data, sending a retransmission instruction to the sending device, includes: Compare the segment numbers obtained within the preset time period with the pre-stored segment sequences; If the segment sequence number obtained within the preset time period does not match the pre-stored segment sequence, the lost segment sequence number is determined. A retransmission instruction is sent to the sending device, wherein the retransmission instruction includes at least the segment sequence number.

9. The cross-network data exchange method according to claim 1, characterized in that, The method further includes: Based on the timestamp, determine whether the transmitted data is duplicate data.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can implement the cross-network data exchange method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, characterized in that, when the program is executed by a processor, it can implement the cross-network data exchange method according to any one of claims 1-7.

12. A computer program product, said computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it can implement the cross-network data exchange method described in any one of claims 1-7.

Images