Method for efficiently collecting, transmitting and processing data in complex closed environment

Abstract

The invention relates to the technical field of data processing, and particularly discloses a method for efficiently acquiring, transmitting and processing data in a complex closed environment, which comprises the following steps of: designing a multi-source data acquisition management module aiming at the problem of low acquisition efficiency of multi-source data with different sizes in a wireless network without an ACK (Acknowledgement Character) mechanism in the complex closed environment; aiming at the problem of data transmission failure caused by instability of a wireless network, designing a breakpoint resume framework; aiming at the problems of low management efficiency of multi-source data element information, data transmission correctness in a complex closed weak network environment, a wireless network without an ACK (Acknowledgement Character) mechanism and the like, a data interaction framework of an acquisition node and a receiving node, a data integrity verification processing framework and an interaction framework between the acquisition node and the receiving node are respectively designed. According to the efficient data acquisition, transmission and processing method in the complex closed environment, the integrity and accuracy of data transmission in a weak network environment are guaranteed through a data integrity verification processing framework, and when a data slice of a receiving end is missing, a missing fragment can be quickly positioned.

Description

Technical Field

[0001] The present invention relates to the technical fields of data acquisition, transmission, and processing, and in particular, to a method for efficiently acquiring, transmitting, and processing data in a complex enclosed environment. Background Art

[0002] In a wireless weak network environment, for multi-source data of different sizes, the traditional method uses a mixed acquisition method of large files and small files, which will lead to problems such as low data acquisition and transmission efficiency and insufficient utilization of weak network bandwidth.

[0003] In complex enclosed environments such as tunnels, due to the narrow space, wireless signals will face problems such as multipath reflection and ground absorption in data transmission, resulting in a shortened wireless signal transmission distance and a high data loss rate, which poses more special requirements for the data transmission system; at the same time, retransmission of large files after interruption in a weak network environment will cause the transmission time to increase exponentially, not only reducing the data transmission efficiency, but also repeatedly occupying the system network resources.

[0004] In addition, in a wireless weak network environment without ACK (Acknowledgment), it is impossible to determine whether the information is sent successfully and whether the data is complete and accurate. ACK plays a crucial role in network communication. ACK means that the receiving party has successfully received the data packet sent by the sending party and is ready to receive the next data packet. If the sending party does not receive the ACK signal within a certain period of time, it will assume that the data packet may be lost or in error, and then retransmit the data to avoid data loss. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for efficiently acquiring, transmitting, and processing data in a complex enclosed environment, and designs a multi-source data acquisition management module, a breakpoint resumption framework, a complete verification processing framework, and an interaction framework between the acquisition node and the receiving node, so as to realize the efficient acquisition, transmission, and processing of multi-source data in a weak network environment, and ensure that the data reaches the receiving node from the acquisition node completely and efficiently.

[0006] To achieve the above purpose, the present invention provides a method for efficiently acquiring, transmitting, and processing data in a complex enclosed environment, including the following steps:

[0007] S1. Design a multi-source data acquisition management module in a wireless network without an ACK mechanism in a complex enclosed environment;

[0008] S2. Design a breakpoint resumption framework;

[0009] S3. Design a data complete verification processing framework;

[0010] S4. Design an interaction framework between the acquisition node and the receiving node.

[0011] Preferably, in S1, small files transmitted using the instant transmission mechanism and large files transmitted using slice multi-thread parallel transmission are separated through threshold setting, and the collected file-related information is stored in the relational database SendData.sql of the collection node. The relational database SendData.sql of the collection node is responsible for interacting with the collection node and storing the collected multi-source data information.

[0012] Preferably, the relational database SendData.sql of the collection node includes data tables send_file_info, send_slice_info, big_file_info, small_file_info, and breakpoint_info;

[0013] Among them, small file information is stored in the small_file_info data table, large file information is stored in the big_file_info data table, sent file information is stored in the send_file_info data table, sent large file slice information is stored in the send_slice_info data table, and large file breakpoint information is stored in the breakpoint_info data table.

[0014] Preferably, the data table small_file_info contains two fields: the sent small file name send_small_file_name and the small file sending success flag send_small_file_result;

[0015] The big_file_info table contains two fields: the sent large file name send_big_file_name and the large file sending success flag send_big_file_result;

[0016] The send_file_info table contains five fields: the sent file name send_file_name, the total number of sent file slices send_slice_total, the sent file hash value send_file_hash, the sent file size send_file_size, and the file sending success flag send_file_result;

[0017] The send_slice_info table contains five fields: the sent file name send_file_name, the sent file slice number send_slice_number, the sent slice hash value send_slice_hash, the sent slice size send_slice_size, and the slice sending success flag slice_send_result;

[0018] The breakpoint_info table contains two fields: the file name file_name and the breakpoint slice number breakpoint_slice_number.

[0019] Preferably, in S2, design a breakpoint resumption framework, which specifically includes the following steps:

[0020] S21. Traverse the source file directory, detect whether the currently traversed file is successfully transmitted, obtain the file meta-information from the relational database of the acquisition node through the interaction framework between the acquisition node and the database. If the file send success flag is false, then send the file meta-information;

[0021] S22. Detect whether the slice files are successfully transmitted in a multi-threaded parallel manner. If the slice file send success flag is false, then send the slice meta-information, and obtain the file breakpoint information from the relational database of the acquisition node, and transmit the breakpoint slice data in a multi-threaded parallel manner. For each successfully sent slice, update the slice send success flag in the relational database of the acquisition node to true;

[0022] S23. Determine whether there are remaining slices. If so, continue to detect whether the slice files are successfully transmitted in a multi-threaded parallel manner, and loop like this until all slice data is sent;

[0023] S24. Send the data transmission completion information, and update the file send success flag in the relational database of the acquisition node to true.

[0024] Preferably, in S3, design a data integrity verification and processing framework, which specifically includes the following steps:

[0025] S31. The acquisition end obtains the fingerprint of the source file, and then sends the data meta-information;

[0026] S32. Slice the obtained source file, obtain the fingerprint of each slice file, and transmit the slice meta-information and slice data in a multi-threaded parallel manner;

[0027] S33. The receiving end processes the received slice data in parallel, verifies whether the slice file is correct. If it is correct, update the slice reception flag of the receiving node database RecvData.sql, receive the data transmission completion information, and then enter S34; if it is incorrect, return to S32 and continue to transmit the slice meta-information and slice data in a multi-threaded parallel manner;

[0028] S34. The receiving end combines and verifies all received data. If the data is correct, it updates the data reception flag in the receiving node database RecvData.sql; if not, it queries the missing slice data in the database, sends the information of the missing slice data to the acquisition end for data element information management, and then enters S32 to transmit the slice meta-information and slice data in a multi-threaded parallel manner again.

[0029] Preferably, in S33, the receiving node database RecvData.sql is responsible for interacting with the receiving node and storing the received multi-source data information. The receiving node database RecvData.sql includes data tables recv_file_info and recv_slice_info;

[0030] Among them, the received file information is stored in the recv_file_info data table, and the received slice information is stored in the recv_slice_info data table;

[0031] The recv_slice_info data table contains five fields: the received file name recv_file_name, the received file slice serial number recv_slice_number, the received slice hash value recv_slice_hash, the received slice size recv_slice_size, and the slice reception success flag recv_slice_result;

[0032] The recv_file_info data table contains five fields: the primary key of the sent file name recv_file_name, the total number of sent file slices recv_slice_total, the received file hash value recv_file_hash, the received file size recv_file_size, and the file reception success flag recv_file_result.

[0033] Preferably, in S33, the receiving end processes the received slice data in parallel to verify whether the slice file is correct. The specific process is as follows:

[0034] Using the MD5 information digest algorithm, perform a hash operation on the received slice data and obtain the hash value of the slice data; compare the calculated slice hash value with the slice fingerprint information recv_slice_hash of the original data saved in the received slice information recv_slice_info table;

[0035] If the two values are the same, it indicates that the transmitted slice data is complete and error-free; otherwise, it indicates that the slice data is transmitted incorrectly, and the slice data is retransmitted.

[0036] Preferably, in S34, the receiving end combines and verifies all received data. The specific process is as follows:

[0037] After all slice data is received, combine them in the slice encoding order, and use the MD5 information digest algorithm to obtain the hash value of the overall data after slice combination; compare the calculated hash value of the overall data with the fingerprint information recv_file_hash of the original data saved in the recv_file_info table;

[0038] If the two values are the same, it indicates that the transmitted overall data is complete and error-free, and update the data reception success flag recv_file_result in the recv_file_info table to true. Otherwise, query the missing slice data information in the database and feedback it to the acquisition end in real time for data slice retransmission.

[0039] Preferably, in S4, the interaction process between the acquisition node and the receiving node is designed as follows: The specific steps are as follows:

[0040] S41. The acquisition node sends data element information to the receiving node. The receiving node stores the data element information in the recv_file_info table in the receiving database RecvData.sql and replies with a data element information response message; the acquisition node determines whether to resend the data element information according to the reception success flag in the reply message. If the flag is true, go to S42; otherwise, resend the data element information;

[0041] S42. The acquisition node sends slice element information to the receiving node. The receiving node stores the slice element information in the recv_slice_info table in the receiving database RecvData.sql and replies with a slice element information response message; the acquisition node determines whether to resend the slice element information according to the reception success flag in the reply message. If the flag is true, go to S43; otherwise, resend the slice element information;

[0042] S43. The acquisition node sends slice data to the receiving node. The receiving node stores the slice data in a temporary file in the local directory and updates the slice reception success flag recv_slice_result in the recv_slice_info table to true; then locate and combine according to the slice number and slice size;

[0043] Finally, the receiving node replies with a slice data response message. The acquisition node determines whether to resend the slice data according to the reception success flag in the reply message. If the flag is true, go to S44; otherwise, resend the slice data;

[0044] S44, the collecting node sends data transmission completion information to the receiving node, and the receiving node determines whether each fragment of the data has been successfully received by detecting whether the check flag bit of each fragment is true;

[0045] If there is no missing slice data, the receiving node replies with data transmission completion verification information, updates the file reception success flag recv_file_result in the recv_file_info table to true, and stores the received temporary file in the HDFS distributed file system; if there is missing slice data, the missing data slice information is quickly found, and the receiving node replies with data transmission completion verification information and missing slice data information;

[0046] The collection node determines whether to send the slice data repeatedly based on the verification success flag in the data transmission completion verification reply information. If the verification flag is false, the slice data is resent based on the missing slice data information in the reply. If the verification flag is true, the file sending success flag send_file_result in the send_file_info table is updated to true.

[0047] Therefore, the present invention adopts the above-mentioned method for efficient data collection, transmission and processing in a complex and closed environment, and the beneficial effects are as follows:

[0048] (1) Aiming at the problem of low efficiency in multi-source data collection of different sizes, the present invention designs a multi-source data collection management module, separates small files transmitted using the second transmission mechanism and large files transmitted using slice multi-threaded parallel transmission through threshold setting, and stores the collected file-related information in the collection node SendData.sql relational database.

[0049] (2) The present invention aims to solve the problem of data transmission failure caused by unstable wireless network in complex, closed and weak network environment. By designing a breakpoint resume framework and a breakpoint information table, the present invention realizes automatic positioning and stable transmission of interrupted data, thereby solving the problems of exponentially increased transmission time due to data retransmission, low transmission efficiency, and repeated occupation of system network resources.

[0050] (3) The present invention aims at the problem of correctness of data transmission in complex, closed and weak network environments, and ensures the integrity and accuracy of data transmission in weak network environments through the designed data integrity verification processing framework. The MD5 information digest algorithm is used to obtain the MD5 hash value of the received slice data and the overall data, and the original slice fingerprint information and the fingerprint information of the original data are compared, so as to quickly and accurately realize the integrity verification of the data. At the same time, the incomplete retransmission mechanism and the missing slice fast positioning mechanism ensure the correctness of data transmission.

[0051] (4) In a wireless network without an ACK (acknowledgment character) mechanism in a complex enclosed environment, the sending end cannot determine whether the data has successfully reached the receiving end, resulting in data loss. To address this problem, the present invention designs a system data interaction UML sequence diagram and data interaction format between the acquisition end and the receiving end to ensure the successful transmission of information in a wireless network environment without ACK.

[0052] The following will further describe the technical solutions of the present invention in detail through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a multi-source data acquisition management flowchart of an embodiment of a method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to the present invention;

[0054] Figure 2 is a breakpoint resumption framework flowchart of an embodiment of a method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to the present invention;

[0055] Figure 3 is a complete verification framework flowchart of an embodiment of a method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to the present invention;

[0056] Figure 4 is an acquisition node and database interaction framework flowchart of an embodiment of a method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to the present invention;

[0057] Figure 5 is a schematic diagram of the interaction framework between the receiving node and the database of an embodiment of a method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to the present invention;

[0058] Figure 6 is a schematic diagram of the interaction process between the acquisition node and the receiving node of an embodiment of a method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0059] The following further illustrates the technical solutions of the present invention through the accompanying drawings and embodiments.

[0060] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the ordinary meanings understood by those of ordinary skill in the art to which the present invention pertains.

[0061] As shown in the figure, a method for efficient data acquisition, transmission, and processing in a complex enclosed environment includes the following steps:

[0062] S1. In a wireless network without an ACK mechanism in a complex enclosed environment, design a multi-source data acquisition management module; the multi-source data acquisition management process is as Figure 1As shown, through threshold setting, multi-source data of different sizes are classified to separate small files transmitted using the instant upload mechanism and large files transmitted using slice multi-threading parallel transmission. And the collected file-related information is stored in the relational database SendData.sql of the collection node. The relational database SendData.sql of the collection node is responsible for interacting with the collection node and storing the multi-source data information collected.

[0063] As shown in Table 1, the relational database SendData.sql of the collection node includes data tables send_file_info, send_slice_info, big_file_info, small_file_info, and breakpoint_info;

[0064] Among them, the small file information is stored in the data table small_file_info, the large file information is stored in the data table big_file_info, the sent file information is stored in the data table send_file_info, the sent large file slice information is stored in the data table send_slice_info, and the large file breakpoint information is stored in the data table breakpoint_info.

[0065] Furthermore, the data table small_file_info contains two fields: the sent small file name send_small_file_name and the small file sending success flag send_small_file_result;

[0066] The big_file_info table contains two fields: the sent large file name send_big_file_name and the large file sending success flag send_big_file_result;

[0067] The send_file_info table contains five fields: the sent file name send_file_name, the total number of sent file slices send_slice_total, the sent file hash value send_file_hash, the sent file size send_file_size, and the file sending success flag send_file_result;

[0068] The send_slice_info table contains five fields: the sent file name send_file_name, the sent file slice serial number send_slice_number, the sent slice hash value send_slice_hash, the sent slice size send_slice_size, and the slice sending success flag slice_send_result;

[0069] The breakpoint_info table contains two fields: file name file_name and breakpoint slice number breakpoint_slice_number.

[0070] Table 1 Structure of the relational database table for the acquisition node

[0071]

[0072]

[0073] S2. In a complex enclosed weak network environment, due to the instability of the wireless network, data transmission failures may occur. To address this issue, a breakpoint resumption framework is designed, as Figure 2 shown. This breakpoint resumption framework can ensure that large files in a weak network environment can resume transmission from the breakpoint after being interrupted, and solves problems such as the transmission time increasing exponentially due to data retransmission, the reduction of transmission efficiency, and the repeated occupation of system network resources.

[0074] The design of the breakpoint resumption framework specifically includes the following steps:

[0075] S21. Traverse the source file directory, detect whether the currently traversed file has been successfully transmitted, and obtain the file meta-information from the relational database of the acquisition node through the interaction framework between the acquisition node and the database (as Figure 4 shown). If the file send success flag is false, then send the file meta-information;

[0076] S22. Detect whether the slice file has been successfully transmitted in a multi-threaded parallel manner. If the slice file send success flag is false, then send the slice meta-information, and obtain the file breakpoint information from the relational database of the acquisition node, and transmit the breakpoint slice data in a multi-threaded parallel manner. For each successfully sent slice, update the slice send success flag in the relational database of the acquisition node to true;

[0077] S23. Determine whether there are remaining slices. If so, continue to detect whether the slice file has been successfully transmitted in a multi-threaded parallel manner, and loop like this until all slice data has been sent;

[0078] S24. Send the data transmission completion information, and update the file send success flag in the relational database of the acquisition node to true.

[0079] Through this method, the automatic positioning and stable transmission of interrupted data are achieved, and the data transmission efficiency is improved.

[0080] S3. Design a data integrity verification processing framework, as Figure 3As shown in the figure, through the data integrity verification technology, the integrity and accuracy of data transmission in a weak network environment are guaranteed. When data slices are missing, the missing segments can be quickly located.

[0081] Design a data integrity verification processing framework, which specifically includes the following steps:

[0082] S31. The acquisition end obtains the fingerprint of the source file and then sends the data element information;

[0083] S32. Slice the obtained source file, obtain the fingerprint of each slice file, and transmit the slice metadata and slice data in a multi-threaded parallel manner;

[0084] S33. The receiving end processes the received slice data in parallel to verify whether the slice file is correct. If it is correct, update the slice reception flag in the receiving node database RecvData.sql, receive the data transmission completion information, and then enter S34; if it is incorrect, return to S32 and continue to transmit the slice metadata and slice data in a multi-threaded parallel manner;

[0085] The receiving node database RecvData.sql is responsible for interacting with the receiving node and storing the received multi-source data information. The receiving node database RecvData.sql includes data tables recv_file_info and recv_slice_info;

[0086] Among them, the received file information is stored in the recv_file_info data table, and the received slice information is stored in the recv_slice_info data table;

[0087] The recv_slice_info data table contains five fields: received file name recv_file_name, received file slice number recv_slice_number, received slice hash value recv_slice_hash, received slice size recv_slice_size, and slice reception success flag recv_slice_result;

[0088] The recv_file_info data table contains five fields: received file name recv_file_name primary key, total number of received file slices recv_slice_total, received file hash value recv_file_hash, received file size recv_file_size, and file reception success flag recv_file_result.

[0089] Regarding the correctness problem of slice data transmission, the receiving end processes the received slice data in parallel to verify whether the slice file is correct. The specific process is as follows:

[0090] Using the MD5 message-digest algorithm, this algorithm performs a hash operation on the received slice data and obtains the hash value of the slice data; compares the calculated slice hash value with the slice fingerprint information recv_slice_hash of the original data saved in the received slice information recv_slice_info table;

[0091] If the two values are the same, it indicates that the transmitted slice data is complete and error-free; otherwise, it indicates that the slice data transmission is incorrect, and the slice data is retransmitted.

[0092] S34. The receiving end combines and verifies all received data. If the data is correct, it updates the data reception flag in the receiving node database RecvData.sql; if it is incorrect, it queries the missing slice data in the database, sends the missing slice data information to the acquisition end for data element information management, and then enters S32 to re-transmit the slice meta-information and slice data in a multi-threaded parallel manner.

[0093] Regarding the correctness issue of the overall data transmission, the receiving end combines and verifies all received data. The specific process is as follows:

[0094] When all slice data is received, it needs to be combined in the slice encoding order, and the MD5 message-digest algorithm is used to obtain the hash value of the overall data after the slices are combined; compares the calculated hash value of the overall data with the fingerprint information recv_file_hash of the original data saved in the recv_file_info table;

[0095] If the two values are the same, it indicates that the transmitted overall data is complete and error-free, and updates the data reception success flag recv_file_result in the recv_file_info table to true; otherwise, it queries the missing slice data information in the database and real-time feedbacks it to the acquisition end for data slice retransmission.

[0096] S4. In a wireless network in a complex airtight environment without an ACK (acknowledgment character) mechanism, design the interaction framework between the acquisition node and the receiving node, that is, as Figure 6 shown, design the system data interaction UML sequence diagram and data interaction format between the acquisition node and the receiving node to ensure that information can be successfully sent in an ACK-free network environment.

[0097] In S4, the designed interaction process between the acquisition node and the receiving node is as follows: The specific steps are as follows:

[0098] S41. The acquisition node sends data element information to the receiving node. The receiving node stores the data element information in the recv_file_info table in the receiving database RecvData.sql, as shown in Table 2, and then replies with a data element information response message. The acquisition node determines whether to resend the data element information according to the success flag in the reply message. If the flag is true, go to S42; otherwise, resend the data element information.

[0099] Table 2 Structure Diagram of the Receiving Node Database Table

[0100]

[0101] S42. The acquisition node sends slice element information to the receiving node. The receiving node stores the slice element information in the recv_slice_info table in the receiving database RecvData.sql, as shown in Table 2, and then replies with a slice element information response message. The acquisition node determines whether to resend the slice element information according to the success flag in the reply message. If the flag is true, go to S43; otherwise, resend the slice element information.

[0102] S43. The acquisition node sends slice data to the receiving node. The receiving node stores the slice data in a temporary file in the local directory (directory naming rule: time - project number - location), and updates the slice reception success flag recv_slice_result in the recv_slice_info table to true; then locates and combines according to the slice number and slice size.

[0103] Finally, the receiving node replies with a slice data response message. The acquisition node determines whether to resend the slice data according to the success flag in the reply message. If the flag is true, go to S44; otherwise, resend the slice data.

[0104] S44. The acquisition node sends data transfer completion information to the receiving node. The receiving node determines whether each fragment of the data has been successfully received by checking whether each fragment verification flag is true.

[0105] If there is no missing slice data, the receiving node replies with data transfer completion verification information, updates the file reception success flag recv_file_result in the recv_file_info table to true, and stores the received temporary file in the HDFS distributed file system.

[0106] If there is missing slice data, quickly find the missing data slice information. The receiving node replies with data transfer completion verification information and missing slice data information.

[0107] The acquisition node determines whether to resend the slice data according to the success flag in the verification reply information for data transmission. If the verification flag is false, it resends the slice data according to the missing slice data information in the reply. If the verification flag is true, it updates the file transmission success flag send_file_result in the send_file_info table to true.

[0108] In this embodiment, the data transmission format at the acquisition end is as follows:

[0109] The sent data element information: data integrity information type, file name, file size, total number of slices, overall file hash value.

[0110] The sent slice meta information: slice meta information type, file name, slice number, slice size, slice hash value.

[0111] The sent slice data: slice data type, file name, slice number, slice size, slice data.

[0112] The data transmission completion information: data transmission completion information type, file name, completion flag bit.

[0113] In this embodiment, the data transmission format at the receiving end is as follows:

[0114] The data element information response: data integrity information response type, file name, success flag for reception.

[0115] The slice meta information response: slice meta information response type, file name, slice number, success flag for reception.

[0116] The slice data response: slice data response type, file name, slice number, success flag for reception.

[0117] The data transmission completion verification reply information: data transmission completion response type, file name, success flag for verification.

[0118] The missing slice data information: missing slice data request type, file name, total number of slices, retransmission slice number, slice size, whether to retransmit slice meta information.

[0119] To distinguish different data transmission formats, the data transmission types at the acquisition end and the receiving end are also respectively defined:

[0120] The data transmission types at the acquisition end: data integrity information type, slice meta information type, slice data type, data transmission completion information type.

[0121] Data transfer types at the receiving end: data integrity information response type, slice meta - information response type, slice data response type, data transfer completion response type, missing slice data request type.

[0122] Meanwhile, to achieve efficient data management, the acquisition node database SendData.sql as shown in Table 1 and the receiving node database RecvData.sql as shown in Table 2 are designed respectively.

[0123] Therefore, the present invention adopts the above - mentioned method for efficient data acquisition, transmission, and processing in a complex airtight environment. Through the designed multi - source data acquisition management module, the efficiency of multi - source data acquisition is improved; through the designed breakpoint - resumed transmission and data integrity verification processing framework, the efficiency, integrity, and accuracy of data transmission are ensured in a complex airtight and weak - network environment; through the designed system interaction framework between the acquisition node and the receiving node, the correctness of data transmission is guaranteed in a wireless network environment without an ACK mechanism; through the designed system database interaction framework between the acquisition node and the receiving node, the efficient management of data meta - information of the acquisition node and the receiving node is realized respectively.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that they can still modify or equivalently replace the technical solutions of the present invention, and these modifications or equivalent replacements do not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for efficient data acquisition, transmission, and processing in a complex closed environment, characterized in that It includes the following steps: S1. Design a multi-source data acquisition and management module in a wireless network without an ACK mechanism in a complex closed environment; S2. Design a breakpoint resumption framework; S3. Design a data integrity verification and processing framework; S4. Design an interaction framework between the acquisition node and the receiving node.

2. The method for efficiently collecting, transmitting, and processing data in a complex enclosed environment according to claim 1, wherein In S1, small files transmitted using the instant transmission mechanism and large files transmitted using slice multi-thread parallel transmission are separated through threshold setting, and the file-related information collected is stored in the relational database SendData.sql of the acquisition node. The relational database SendData.sql of the acquisition node is responsible for interacting with the acquisition node and storing the multi-source data information collected.

3. A method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to claim 2, characterized in that, The relational database SendData.sql of the acquisition node includes data tables send_file_info, send_slice_info, big_file_info, small_file_info, and breakpoint_info; Among them, the small file information is stored in the data table small_file_info, the large file information is stored in the data table big_file_info, the sent file information is stored in the data table send_file_info, the sent large file slice information is stored in the data table send_slice_info, and the large file breakpoint information is stored in the data table breakpoint_info.

4. A method for efficient data acquisition, transmission, and processing in a complex closed environment according to claim 3, characterized in that The data table small_file_info contains two fields: the sent small file name send_small_file_name and the small file sending success flag send_small_file_result; The big_file_info table contains two fields: the sent large file name send_big_file_name and the large file sending success flag send_big_file_result; The send_file_info table contains five fields: the sent file name send_file_name, the total number of sent file slices send_slice_total, the sent file hash value send_file_hash, the sent file size send_file_size, and the file sending success flag send_file_result; The send_slice_info table contains five fields: the sent file name send_file_name, the sent file slice serial number send_slice_number, the sent slice hash value send_slice_hash, the sent slice size send_slice_size, and the slice sending success flag slice_send_result; The breakpoint_info table contains two fields: the file name file_name and the breakpoint slice number breakpoint_slice_number.

5. A method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to claim 4, characterized in that, In S2, design a breakpoint resumption framework, which specifically includes the following steps: S21. Traverse the source file directory, detect whether the currently traversed file has been successfully transmitted, obtain the file meta-information from the relational database of the acquisition node through the interaction framework between the acquisition node and the database. If the file transmission success flag is false, then send the file meta-information. S22. Detect whether the sliced files have been successfully transmitted in a multi-threaded parallel manner. If the sliced file transmission success flag is false, then send the sliced meta-information, obtain the file breakpoint information from the relational database of the acquisition node, and transmit the breakpoint sliced data in a multi-threaded parallel manner. For each successfully transmitted slice, update the slice transmission success flag in the relational database of the acquisition node to true. S23. Determine whether there are remaining slices. If so, continue to detect whether the sliced files have been successfully transmitted in a multi-threaded parallel manner. Repeat this loop until all sliced data has been sent. S24. Send the data transmission completion information, and update the file transmission success flag in the relational database of the acquisition node to true.

6. A method for efficient data acquisition, transmission, and processing in a complex closed environment according to claim 5, characterized in that In S3, design a data integrity verification processing framework, which specifically includes the following steps: S31. The acquisition end obtains the fingerprint of the source file, and then sends the data meta-information. S32. Slice the obtained source file, obtain the fingerprint of each sliced file, and transmit the sliced meta-information and sliced data in a multi-threaded parallel manner. S33. The receiving end processes the received sliced data in parallel, verifies whether the sliced files are correct. If correct, update the slice reception flag in the receiving node database RecvData.sql, receive the data transmission completion information, and then proceed to S34. If incorrect, return to S32 and continue to transmit the sliced meta-information and sliced data in a multi-threaded parallel manner. S34. The receiving end combines and verifies all received data. If the data is correct, update the data reception flag in the receiving node database RecvData.sql. If incorrect, query the missing sliced data in the database, send the missing sliced data information to the acquisition end for data meta-information management, and then enter S32 to transmit the sliced meta-information and sliced data in a multi-threaded parallel manner again.

7. A method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to claim 6, characterized in that In S33, the receiving node database RecvData.sql is responsible for interacting with the receiving node and storing the received multi-source data information. The receiving node database RecvData.sql includes data tables recv_file_info and recv_slice_info. Among them, the received file information is stored in the recv_file_info data table, and the received slice information is stored in the recv_slice_info data table. The recv_slice_info data table contains five fields: received file name recv_file_name, received file slice number recv_slice_number, received slice hash value recv_slice_hash, received slice size recv_slice_size, and slice reception success flag recv_slice_result. The recv_file_info data table contains five fields: the primary key of the sent file name recv_file_name, the total number of slices of the sent file recv_slice_total, the hash value of the received file recv_file_hash, the size of the received file recv_file_size, and the file reception success flag recv_file_result.

8. A method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to claim 7, characterized in that In S33, the receiving end processes the received slice data in parallel to verify whether the slice file is correct. The specific process is as follows: Using the MD5 message digest algorithm, perform a hash operation on the received slice data and obtain the hash value of the slice data; compare the calculated slice hash value with the slice fingerprint information recv_slice_hash of the original data saved in the recv_slice_info table of the received slice information. If the two values are the same, it indicates that the transmitted slice data is complete and error-free; otherwise, it indicates that the slice data transmission is incorrect, and the slice data is retransmitted.

9. A method for efficient data acquisition, transmission, and processing in a complex enclosed environment according to claim 8, characterized in that, In S34, the receiving end combines and verifies all received data. The specific process is as follows: When all slice data is received, combine them in the order of slice encoding, and use the MD5 message digest algorithm to obtain the hash value of the overall data after combining the slices; compare the calculated hash value of the overall data with the fingerprint information recv_file_hash of the original data saved in the recv_file_info table. If the two values are the same, it indicates that the transmitted overall data is complete and error-free, and update the data reception success flag recv_file_result in the recv_file_info table to true; otherwise, query the missing slice data information in the database and feedback it to the acquisition end in real time for data slice retransmission.

10. A method for efficient data acquisition, transmission, and processing in a complex closed environment according to claim 9, characterized in that, In S4, the interaction process between the acquisition node and the receiving node is designed as follows: The specific steps are as follows: S41. The acquisition node sends data element information to the receiving node. The receiving node stores the data element information in the recv_file_info table in the receiving database RecvData.sql and replies with the data element information response message; the acquisition node determines whether to resend the data element information according to the reception success flag in the reply message. If the flag is true, enter S42; otherwise, resend the data element information. S42. The acquisition node sends slice element information to the receiving node. The receiving node stores the slice element information in the recv_slice_info table in the receiving database RecvData.sql and replies with the slice element information response message. The acquisition node determines whether to resend the slice element information according to the reception success flag in the reply message. If the flag is true, enter S43; otherwise, resend the slice element information. S43. The acquisition node sends the sliced data to the receiving node. The receiving node stores the sliced data in a temporary file in the local directory and updates the sliced data reception success flag recv_slice_result in the recv_slice_info table to true. Then, it locates and combines based on the slice number and slice size. Finally, the receiving node replies with the sliced data response information. The acquisition node determines whether to resend the sliced data based on the reception success flag in the reply information. If the flag is true, it proceeds to S44; otherwise, it resends the sliced data. S44. The acquisition node sends the data transmission completion information to the receiving node. The receiving node determines whether each segment of the data has been successfully received by checking whether each segment verification flag is true. If there is no missing sliced data, the receiving node replies with the data transmission completion verification information, updates the file reception success flag recv_file_result in the recv_file_info table to true, and stores the received temporary file in the HDFS distributed file system. If there is missing sliced data, it quickly locates the missing data slice information, and the receiving node replies with the data transmission completion verification information and the missing sliced data information. The acquisition node determines whether to resend the sliced data based on the verification success flag in the data transmission completion verification reply information. If the verification flag is false, it resends the sliced data according to the replied missing sliced data information. If the verification flag is true, it updates the file transmission success flag send_file_result in the send_file_info table to true.

Images