A method for efficient data acquisition, transmission and processing in a complex closed environment

Abstract

The application relates to the technical field of data processing, and particularly discloses a kind of complex closed environment data efficient collection, transmission, processing method, including in the wireless network of complex closed environment no ACK mechanism, for the low efficiency problem of different size of multi-source data collection, design multi-source data collection management module;For the data transmission failure problem caused by unstable wireless network, design breakpoint resume transmission framework;For the low efficiency of multi-source data element information management, the correctness of complex closed weak network environment data transmission and the problem of no ACK mechanism wireless network, respectively design the data interaction framework of collection node and receiving node, data integrity check processing framework and the interaction framework between the two.The application adopts the above-mentioned kind of complex closed environment data efficient collection, transmission, processing method, through data integrity check processing framework, guarantee the integrity, accuracy of data transmission under weak network environment, when receiving end data slice is missing, can quickly locate the missing fragment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, transmission and processing, in particular to a method for efficient data collection, transmission and processing in a complex closed environment. BACKGROUND

[0002] In a wireless weak network environment, for multi-source data of different sizes, the traditional method adopts a mixed collection method of large files and small files, which can cause problems such as low data collection and transmission efficiency, insufficient weak network bandwidth utilization, etc.

[0003] In a complex closed environment such as a tunnel, due to the small space, wireless signals will face problems such as multipath reflection and ground absorption in data transmission, resulting in a shortened wireless signal transmission distance and a high data loss rate, which puts forward more special requirements for the data transmission system; at the same time, retransmission of large files after interruption in a weak network environment will cause the transmission time to increase exponentially, not only reducing the data transmission efficiency, but also repeatedly occupying system network resources.

[0004] In addition, in a wireless weak network environment without ACK (Acknowledgment), it is impossible to determine whether the information is successfully sent and whether the data is complete and accurate. ACK plays a crucial role in network communication, indicating that the receiving party has successfully received the data packet from the sending party and is ready to receive the next data packet. If the sender does not receive an ACK signal within a certain time, it will consider that the data packet may be lost or have errors, and then repeatedly send the data to avoid data loss. SUMMARY

[0005] The purpose of the present application is to provide a method for efficient data collection, transmission and processing in a complex closed environment, which designs a multi-source data collection management module, a breakpoint resume transmission framework, a complete verification processing framework, an interaction framework between the collection node and the receiving node, realizes efficient collection, transmission and processing of multi-source data in a weak network environment, and ensures that the data is complete and efficiently reaches the receiving node from the collection node.

[0006] To achieve the above purpose, the present application provides a method for efficient data collection, transmission and processing in a complex closed environment, comprising the following steps:

[0007] S1. In a wireless network without ACK mechanism in a complex closed environment, a multi-source data collection management module is designed;

[0008] S2. A breakpoint resume transmission framework is designed;

[0009] S3. A data complete verification processing framework is designed;

[0010] S4. An interaction framework between the collection node and the receiving node is designed.

[0011] Preferably, in S1, small files transmitted using the second sharing mechanism and large files transmitted using the slice multi-thread parallel transmission are separated by threshold setting, and the collected file related information is stored in the collection node SendData.sql relational database. The collection node SendData.sql relational database is responsible for interaction with the collection node and storage of the collected multi-source data information.

[0012] Preferably, the collection node SendData.sql relational database includes data tables send_file_info, send_slice_info, big_file_info, small_file_info and breakpoint_info.

[0013] Among them, small file information is stored in the small_file_info data table, large file information is stored in the big_file_info data table, sending file information is stored in the send_file_info data table, large file slice information sent is stored in the send_slice_info data table, and large file breakpoint information is stored in the breakpoint_info data table.

[0014] Preferably, the data table small_file_info contains two fields of sending small file name send_small_file_name and small file sending success identification send_small_file_result.

[0015] The big_file_info table contains two fields of sending large file name send_big_file_name and large file sending success identification send_big_file_result.

[0016] The send_file_info table contains five fields of sending file name send_file_name, sending file slice total send_slice_total, sending file hash value send_file_hash, sending file size send_file_size and file sending success identification send_file_result.

[0017] The send_slice_info table contains five fields of sending file name send_file_name, sending file slice serial number send_slice_number, sending slice hash value send_slice_hash, sending slice size send_slice_size and slice sending success identification slice_send_result.

[0018] The breakpoint_info table includes two fields: a file name file_name and a breakpoint slice number breakpoint_slice_number.

[0019] Preferably, in S2, a breakpoint resuming framework is designed, and the framework specifically includes the following steps:

[0020] S21, files in a source file directory are traversed to detect whether a currently traversed file is successfully transmitted, file meta information is obtained from a collecting node relational database through a collecting node and database interaction framework, and if the file sending success flag is false, the file meta information is sent;

[0021] S22, whether the slice file is successfully transmitted is detected in a multi-thread parallel manner, if the slice file sending success flag is false, the slice meta information is sent, and file breakpoint information is obtained from the collecting node relational database, the breakpoint slice data is transmitted in a multi-thread parallel manner, the slice sending success flag in the collecting node relational database is updated to true every time a slice is successfully sent;

[0022] S23, whether there is a remaining slice is judged, if there is, the slice file is continuously detected in a multi-thread parallel manner, and the cycle is continued until all the slice data is sent;

[0023] S24, data transmission completion information is sent, and the file sending success flag in the collecting node relational database is updated to true.

[0024] Preferably, in S3, a data integrity check processing framework is designed, and the framework specifically includes the following steps:

[0025] S31, a fingerprint of a source file is obtained by a collecting end, and then data meta information is sent;

[0026] S32, the source file is sliced to obtain a fingerprint of each slice file, and the slice meta information and slice data are transmitted in a multi-thread parallel manner;

[0027] S33, the received slice data is processed in parallel by a receiving end, whether the slice file is correct is checked, if yes, a slice receiving flag in a receiving node database RecvData.sql is updated, data transmission completion information is received, and then S34 is entered; if no, S32 is returned to continue transmitting the slice meta information and slice data in a multi-thread parallel manner;

[0028] S34, the receiving end combines and checks all received data, if the data is correct, the receiving node database RecvData.sql data receiving identifier is updated; if not, the missing slice data is queried from the database, the missing slice data information is sent to the acquisition end for data element information management, and then S32 is entered to retransmit the slice element information and slice data in a multi-thread parallel manner.

[0029] Preferably, in S33, the receiving node database RecvData.sql is responsible for interacting with the receiving node and storing the received multi-source data information, and the receiving node database RecvData.sql includes data tables recv_file_info and recv_slice_info.

[0030] Among them, the received file information is stored in the recv_file_info data table, and the received slice information is stored in the recv_slice_info data table.

[0031] The recv_slice_info data table includes five fields of received file name recv_file_name, received file slice number recv_slice_number, received slice hash value recv_slice_hash, received slice size recv_slice_size and slice reception success identifier recv_slice_result.

[0032] The recv_file_info data table includes five fields of received file name recv_file_name, received file slice number recv_slice_number, received slice hash value recv_slice_hash, received slice size recv_slice_size and slice reception success identifier recv_slice_result.

[0033] Preferably, in S33, the receiving end processes the received slice data in parallel, checks whether the slice file is correct, and the specific process is as follows:

[0034] The MD5 message digest algorithm is used to perform hash operation on the received slice data, and the hash value of the slice data is obtained; the calculated slice hash value is compared with the slice fingerprint information recv_slice_hash of the original data saved in the received slice information recv_slice_info table.

[0035] If the two values are the same, it means that the transmitted slice data is complete and correct; otherwise, the slice data transmission is incorrect, and the slice data is retransmitted.

[0036] Preferably, in S34, the receiving end combines and checks all received data, and the specific process is as follows:

[0037] When all the slice data is received, the slice data is combined according to the slice encoding order, and the hash value of the combined data is obtained by using the MD5 information digest algorithm; the calculated hash value of the data is compared with the fingerprint information recv_file_hash of the original data saved in the recv_file_info table;

[0038] If the two values are the same, it means that the transmitted overall data is complete and correct, and the data reception success identifier recv_file_result in the recv_file_info table is updated to true, otherwise, the missing slice data information in the database is queried and fed back to the collection end in real time for data slice retransmission.

[0039] Preferably, in S4, the interaction process between the collection node and the receiving node is designed as follows: the specific steps are as follows:

[0040] S41, the collection node sends data element information to the receiving node, the receiving node stores the data element information in the recv_file_info table in the receiving database RecvData.sql, and replies data element information response information; the collection node determines whether to resend the data element information according to the reception success flag in the reply information, if the flag is true, it enters S42, otherwise, it resends the data element information;

[0041] S42, the collection node sends slice element information to the receiving node, the receiving node stores the slice element information in the recv_slice_info table in the receiving database RecvData.sql, and replies slice element information response information, the collection node determines whether to resend the slice element information according to the reception success flag in the reply information, if the flag is true, it enters S43, otherwise, it resends the slice element information;

[0042] S43, the collection node sends slice data to the receiving node, the receiving node stores the slice data in the temporary file in the local directory, and updates the slice reception success identifier recv_slice_result in the recv_slice_info table to true; then, the slice data is positioned and combined according to the slice number and the slice size;

[0043] Finally, the receiving node replies slice data response information, the collection node determines whether to resend the slice data according to the reception success flag in the reply information, if the flag is true, it enters S44, otherwise, it resends the slice data;

[0044] S44, the acquisition node sends data transmission completion information to the receiving node, and the receiving node determines whether each fragment of data has been successfully received by detecting whether each fragment check flag is true or not;

[0045] If there is no missing slice data, the receiving node returns data transmission completion check information, updates the file receiving success identifier recv_file_result in the recv_file_info table to true, and stores the received temporary file in the HDFS distributed file system; if there is missing slice data, the missing data slice information is quickly found, the receiving node returns data transmission completion check information and missing slice data information;

[0046] The acquisition node determines whether to resend slice data according to the whether check success identifier in the data transmission completion check reply information; if the check identifier is false, the slice data is resent according to the returned missing slice data information; if the check identifier is true, the file sending success identifier send_file_result in the send_file_info table is updated to true.

[0047] Therefore, the present application adopts the above-mentioned kind of complex closed environment data efficient acquisition, transmission, processing method, the beneficial effects are as follows:

[0048] (1) The present application is aimed at the low efficiency of multi-source data acquisition of different sizes, and a multi-source data acquisition management module is designed, small files using the P2P mechanism for transmission and large files using slice multi-thread parallel transmission are separated out through threshold setting, and the collected file related information is stored in the acquisition node SendData.sql relational database.

[0049] (2) The present application is aimed at the problem of data transmission failure caused by unstable wireless network in complex closed weak network environment, and through the design of breakpoint resume transmission framework and breakpoint information table, the automatic positioning and stable transmission of interrupted data are realized, and the problems of transmission time increasing exponentially, low transmission efficiency, repeated occupation of system network resources and the like caused by data retransmission are solved.

[0050] (3) The present application is aimed at the correctness of data transmission in complex closed weak network environment, and through the designed data integrity check processing framework, the integrity and accuracy of data transmission in weak network environment are guaranteed. The MD5 hash value of the received slice data and the whole data is obtained by using the MD5 information digest algorithm, and is compared with the original slice fingerprint information and the fingerprint information of the original data, so that the integrity check of the data is quickly and accurately realized. At the same time, the integrity retransmission mechanism and the missing slice fast positioning mechanism ensure the correctness of data transmission.

[0051] (4) in the complex closed environment without ACK (acknowledgement character) mechanism of wireless network, the sending end cannot determine whether the data successfully reaches the receiving end, thereby causing data loss, and the system data interaction UML timing diagram and data interaction format between the collection end and the receiving end are designed to ensure successful information transmission in the ACK-free wireless network environment.

[0052] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a multi-source data collection management flowchart of an embodiment of the data efficient collection, transmission and processing method in the complex closed environment of the application;

[0054] Figure 2 is a breakpoint continuation framework flowchart of an embodiment of the data efficient collection, transmission and processing method in the complex closed environment of the application;

[0055] Figure 3 is a complete verification framework flowchart of an embodiment of the data efficient collection, transmission and processing method in the complex closed environment of the application;

[0056] Figure 4 is a collection node and database interaction framework flowchart of an embodiment of the data efficient collection, transmission and processing method in the complex closed environment of the application;

[0057] Figure 5 is a receiving node and database interaction framework schematic diagram of an embodiment of the data efficient collection, transmission and processing method in the complex closed environment of the application;

[0058] Figure 6 is a collection node and receiving node interaction flow schematic diagram of an embodiment of the data efficient collection, transmission and processing method in the complex closed environment of the application. DETAILED DESCRIPTION

[0059] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments.

[0060] Unless otherwise defined, the technical terms or scientific terms used in the application should be understood as the usual meanings understood by those skilled in the art in the field of the application.

[0061] As shown in the figure, a data efficient collection, transmission and processing method in a complex closed environment comprises the following steps:

[0062] S1, in the complex closed environment without ACK mechanism of wireless network, a multi-source data collection management module is designed; the multi-source data collection management flowchart is as follows Figure 1As shown, by threshold setting, multi-source data of different sizes are classified, small files transmitted using the second sharing mechanism and large files transmitted using the slice multi-thread parallel transmission are separated, and the collected file related information is stored in the collection node SendData.sql relational database. The collection node SendData.sql relational database is responsible for interaction with the collection node and storage of the collected multi-source data information.

[0063] As shown in Table 1, the collection node SendData.sql relational database includes data tables send_file_info, send_slice_info, big_file_info, small_file_info and breakpoint_info.

[0064] Among them, the small file information is stored in the small_file_info data table, the large file information is stored in the big_file_info data table, the sending file information is stored in the send_file_info data table, the sending large file slice information is stored in the send_slice_info data table, and the large file breakpoint information is stored in the breakpoint_info data table.

[0065] Further, the data table small_file_info contains two fields of sending small file name send_small_file_name and small file sending success identification send_small_file_result.

[0066] The big_file_info table contains two fields of sending large file name send_big_file_name and large file sending success identification send_big_file_result.

[0067] The send_file_info table contains five fields of sending file name send_file_name, sending file slice total send_slice_total, sending file hash send_file_hash, sending file size send_file_size and file sending success identification send_file_result.

[0068] The send_slice_info table contains five fields of sending file name send_file_name, sending file slice serial number send_slice_number, sending slice hash send_slice_hash, sending slice size send_slice_size and slice sending success identification slice_send_result.

[0069] The breakpoint_info table includes two fields, a file name file_name and a breakpoint slice number breakpoint_slice_number.

[0070] Table 1: Table structure of the acquisition node relational database

[0071]

[0072]

[0073] S2, in a complex closed weak network environment, data transmission fails due to unstable wireless network. To solve this problem, a breakpoint resume transmission framework is designed, as shown in Figure 2 The breakpoint resume transmission framework can ensure the continuous transmission of large files from the breakpoint after interruption in a weak network environment, and solve the problems of doubling of transmission time, reduction of transmission efficiency, and repeated occupation of system network resources caused by data retransmission.

[0074] The breakpoint resume transmission framework is designed, which specifically includes the following steps:

[0075] S21, traverse the source file directory, detect whether the currently traversed file is transmitted successfully, obtain file meta information from the acquisition node relational database through the acquisition node and database interaction framework (as shown in Figure 4 If the file sending success flag is false, send the file meta information.

[0076] S22, detect whether the slice file is transmitted successfully in a multi-thread parallel manner. If the slice file sending success flag is false, send the slice meta information, and obtain file breakpoint information from the acquisition node relational database. Transmit the breakpoint slice data in a multi-thread parallel manner. Each time a slice is successfully sent, update the slice sending success flag in the acquisition node relational database to true.

[0077] S23, judge whether there is a remaining slice. If there is, continue to detect whether the slice file is transmitted successfully in a multi-thread parallel manner. Repeat the cycle until all slice data is sent.

[0078] S24, send data transmission completion information, and update the file sending success flag in the acquisition node relational database to true.

[0079] Through this method, automatic positioning and stable transmission of interrupted data are realized, and the data transmission efficiency is improved.

[0080] S3, design a data integrity check processing framework, as shown in Figure 3As shown, through the data integrity check technology, the integrity and accuracy of data transmission in a weak network environment are ensured, and when a data slice is missing, the missing slice can be quickly located.

[0081] A data integrity check processing framework is designed, specifically including the following steps:

[0082] S31, the acquisition end acquires the fingerprint of the source file, and then sends the data element information;

[0083] S32, the acquired source file is sliced, the fingerprint of each slice file is acquired, and the slice element information and slice data are transmitted in a multi-thread parallel manner;

[0084] S33, the receiving end processes the received slice data in parallel, checks whether the slice file is correct, if correct, the receiving node database RecvData.sql slice receiving identifier is updated, the receiving data sending completion information is sent, and then S34 is entered; if not correct, return to S32 to continue transmitting the slice element information and slice data in a multi-thread parallel manner;

[0085] The receiving node database RecvData.sql is responsible for interacting with the receiving node and storing the received multi-source data information, and the receiving node database RecvData.sql includes data tables recv_file_info and recv_slice_info;

[0086] Among them, the received file information is stored in the recv_file_info data table, and the received slice information is stored in the recv_slice_info data table;

[0087] The recv_slice_info data table contains five fields of received file name recv_file_name, received file slice number recv_slice_number, received slice hash value recv_slice_hash, received slice size recv_slice_size and slice receiving success identifier recv_slice_result;

[0088] The recv_file_info data table contains five fields of received file name recv_file_name primary key, received file slice total recv_slice_total, received file hash value recv_file_hash, received file size recv_file_size and file receiving success identifier recv_file_result.

[0089] For the correctness of the slice data transmission, the receiving end processes the received slice data in parallel, checks whether the slice file is correct, and the specific process is:

[0090] Using MD5 information digest algorithm, the algorithm by performing hash operation on the received slice data, and obtain the hash value of slice data; the calculated slice hash value and the slice fingerprint information recv_slice_hash saved in the table of received slice information recv_slice_info of the original data are compared;

[0091] If the two values are the same, it means that the transmitted slice data is complete and correct; otherwise, the slice data transmission error, retransmit the slice data.

[0092] S34, the receiving end combines and checks all received data, if the data is correct, update the data receiving identification of the receiving node database RecvData.sql; if not correct, query the missing slice data in the database, send the missing slice data information to the acquisition end for data element information management, and then enter S32 to retransmit the slice element information and slice data in a multi-thread parallel manner.

[0093] For the correctness of the whole data transmission, the receiving end combines and checks all received data, the specific process is:

[0094] When all slice data is received, it needs to be combined according to the slice coding order, and the MD5 information digest algorithm is used to obtain the hash value of the whole data after slice combination; the calculated hash value of the whole data is compared with the fingerprint information recv_file_hash saved in the table of the original data recv_file_info;

[0095] If the two values are the same, it means that the transmitted whole data is complete and correct, and the data receiving success identification recv_file_result in the recv_file_info table is updated to true, otherwise, the missing slice data information in the database is queried, and real-time feedback to the acquisition end for data slice retransmission.

[0096] S4, in the wireless network without ACK (acknowledgement character) mechanism in complex closed environment, the interaction framework between the acquisition node and the receiving node is designed, that is, as shown in Figure 6 The system data interaction UML time sequence diagram and data interaction format between the acquisition node and the receiving node are designed to ensure that the information can be successfully sent in the ACK network environment.

[0097] In S4, the interaction process between the acquisition node and the receiving node is designed, the specific steps are:

[0098] S41, the collection node sends data element information to the receiving node, the receiving node stores the data element information in the recv_file_info table in the receiving database RecvData.sql as shown in Table 2, and replies data element information response information; the collection node determines whether to resend the data element information according to the receiving success flag in the reply information, if the flag is true, it enters S42, otherwise it resends the data element information.

[0099] Table 2: receiving node database table structure diagram

[0100]

[0101] S42, the collection node sends slice element information to the receiving node, the receiving node stores the slice element information in the recv_slice_info table in the receiving database RecvData.sql as shown in Table 2, and replies slice element information response information, the collection node determines whether to resend the slice element information according to the receiving success flag in the reply information, if the flag is true, it enters S43, otherwise it resends the slice element information.

[0102] S43, the collection node sends slice data to the receiving node, the receiving node stores the slice data in the temporary file in the local directory (directory naming rule: time-project number-place), and updates the slice receiving success flag recv_slice_result in the recv_slice_info table to true; then it positions and combines according to the slice number and slice size;

[0103] Finally, the receiving node replies slice data response information, the collection node determines whether to resend the slice data according to the receiving success flag in the reply information, if the flag is true, it enters S44, otherwise it resends the slice data.

[0104] S44, the collection node sends data transmission completion information to the receiving node, the receiving node determines whether each slice has been successfully received by detecting whether the slice check flag is true;

[0105] If there is no missing slice data, the receiving node replies data transmission completion check information, updates the file receiving success flag recv_file_result in the recv_file_info table to true, and stores the received temporary file in the HDFS distributed file system;

[0106] If there is missing slice data, the missing slice information is quickly found, the receiving node replies data transmission completion check information and missing slice data information;

[0107] The collection node determines whether to resend the slice data according to whether the check success identifier in the data transmission completion check reply information, if the check identifier is false, the slice data is resent according to the replied missing slice data information, and if the check identifier is true, a file sending success identifier send_file_result in a send_file_info table is updated to true.

[0108] The data transmission format of the collection end in the embodiment is as follows:

[0109] The sent data element information: data complete information type, file name, file size, total number of slices, and file overall hash value.

[0110] The sent slice element information: slice element information type, file name, slice number, slice size, and slice hash value.

[0111] The sent slice data: slice data type, file name, slice number, slice size, and slice data.

[0112] The data transmission completion information: data transmission completion information type, file name, and completion flag.

[0113] The data transmission format of the receiving end in the embodiment is as follows:

[0114] The data element information response information: data complete information response type, file name, and whether the receiving success flag.

[0115] The slice element information response information: slice element information response type, file name, slice number, and whether the receiving success flag.

[0116] The slice data response information: slice data response type, file name, slice number, and whether the receiving success flag.

[0117] The data transmission completion check reply information: data transmission completion response type, file name, and whether the check success flag.

[0118] The missing slice data information: missing slice data request type, file name, total number of slices, retransmission slice number, slice size, and whether the retransmission slice element information.

[0119] In order to distinguish the formats of different data transmissions, the collection end data transmission type and the receiving end data transmission type are respectively formulated:

[0120] The collection end data transmission type: data complete information type, slice element information type, slice data type, and data transmission completion information type.

[0121] The data transmission type of the receiving end: data complete information response type, slice element information response type, slice data response type, data transmission completion response type, and missing slice data request type.

[0122] In order to realize efficient management of data, the acquisition node database SendData.sql is designed as shown in Table 1 and the receiving node database RecvData.sql is designed as shown in Table 2.

[0123] Therefore, the present application adopts the above-mentioned data efficient acquisition, transmission and processing method in a complex closed environment, improves the multi-source data acquisition efficiency through the designed multi-source data acquisition management module, ensures the efficiency, integrity and accuracy of data transmission in a complex closed weak network environment through the designed breakpoint resume transmission and data integrity verification processing framework, guarantees the correctness of data transmission in a wireless network environment without ACK mechanism through the designed system interaction framework between the acquisition node and the receiving node, and realizes efficient management of data element information of the acquisition node and the receiving node through the designed acquisition node and receiving node system database interaction framework.

[0124] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application rather than limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: the technical solutions of the present application can still be modified or replaced by the equivalent, and these modifications or equivalent replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A method for efficient data acquisition, transmission and processing in a complex closed environment, characterized in that, The method comprises the following steps: S1. Designing a multi-source data collection management module under a wireless network without ACK mechanism in a complex closed environment; S2. Designing a breakpoint resume transmission framework; S3. Designing a data integrity check processing framework; S4. Designing an interaction framework between a collection node and a receiving node, and the interaction process of the interaction framework is specifically as follows: S41. The collection node sends data element information to the receiving node, the receiving node stores the data element information in a recv_file_info table in a receiving database RecvData.sql, and replies to the data element information response information; the collection node determines whether to repeatedly send the data element information according to a receiving success flag in the reply information, if the flag is true, the process goes to S42, otherwise, the data element information is repeatedly sent; S42. The collection node sends slice element information to the receiving node, the receiving node stores the slice element information in a recv_slice_info table in the receiving database RecvData.sql, and replies to the slice element information response information, the collection node determines whether to repeatedly send the slice element information according to a receiving success flag in the reply information, if the flag is true, the process goes to S43, otherwise, the slice element information is repeatedly sent; S43. The collection node sends slice data to the receiving node, the receiving node stores the slice data in a temporary file in a local directory, and updates a slice receiving success flag recv_slice_result in the recv_slice_info table to true; then, the slice data is positioned and combined according to the slice number and the slice size; Finally, the receiving node replies to the slice data response information, the collection node determines whether to repeatedly send the slice data according to a receiving success flag in the reply information, if the flag is true, the process goes to S44, otherwise, the slice data is repeatedly sent; S44. The collection node sends data transmission completion information to the receiving node, the receiving node determines whether the data segments have been successfully received by detecting whether the segment check flags are true; If there is no missing slice data, the receiving node replies to data transmission completion check information, updates a file receiving success flag recv_file_result in the recv_file_info table to true, and stores the received temporary file in an HDFS distributed file system; if there is missing slice data, the missing slice data information is quickly found, the receiving node replies to the data transmission completion check information and the missing slice data information; The collection node determines whether to repeatedly send the slice data according to a check success flag in the data transmission completion check reply information, if the check flag is false, the slice data is re-sent according to the replied missing slice data information, if the check flag is true, a file sending success flag send_file_result in the send_file_info table is updated to true.

2. The method of claim 1, wherein, In S1, small files transmitted by using the second sharing mechanism and large files transmitted by using the slice multi-thread parallel transmission are separated by threshold setting, and the collected file related information is stored in the collection node SendData.sql relational database. The collection node SendData.sql relational database is responsible for interaction with the collection node and storage of the collected multi-source data information.

3. The method of claim 2, wherein the method comprises: The collection node SendData.sql relational database includes data tables send_file_info, send_slice_info, big_file_info, small_file_info and breakpoint_info. Among them, small file information is stored in the small_file_info data table, large file information is stored in the big_file_info data table, sending file information is stored in the send_file_info data table, large file slice information sent is stored in the send_slice_info data table, and large file breakpoint information is stored in the breakpoint_info data table.

4. The method of claim 3, wherein the method comprises: The data table small_file_info contains two fields of sending small file name send_small_file_name and small file sending success identification send_small_file_result. The big_file_info table contains two fields of sending large file name send_big_file_name and large file sending success identification send_big_file_result. The send_file_info table contains five fields of sending file name send_file_name, sending file slice total send_slice_total, sending file hash value send_file_hash, sending file size send_file_size and file sending success identification send_file_result. The send_slice_info table contains five fields of sending file name send_file_name, sending file slice serial number send_slice_number, sending slice hash value send_slice_hash, sending slice size send_slice_size and slice sending success identification slice_send_result. The breakpoint_info table contains two fields of file name file_name and breakpoint slice number breakpoint_slice_number.

5. The method of claim 4, wherein, In S2, a breakpoint resume framework is designed, specifically including the following steps: S21, traverse the source file directory, detect whether the currently traversed file is transmitted successfully, obtain file meta information from the collection node relational database through the collection node and database interaction framework, and if the file sending success identification is false, send the file meta information; S22, detecting whether the slice file is transmitted successfully in a multi-thread parallel manner, if the slice file is transmitted successfully, the slice data is transmitted, and the file breakpoint information is obtained from the relational database of the collection node, and the slice sending success identifier in the relational database of the collection node is updated to true; S23, judging whether there is remaining slice, if yes, the slice file is detected whether transmitted successfully in a multi-thread parallel manner, and the cycle is continued until all the slice data is transmitted; S24, transmitting the data transmission completion information, and updating the file sending success identifier in the relational database of the collection node to true.

6. The method of efficient data acquisition, transmission and processing in a complex closed environment according to claim 5, characterized in that, In S3, the data integrity check processing framework is designed, including the following steps: S31, the fingerprint of the source file is obtained by the collection end, and then the data element information is transmitted; S32, the source file is sliced, the fingerprint of each slice file is obtained, and the slice element information and slice data are transmitted in a multi-thread parallel manner; S33, the received slice data is processed in parallel by the receiving end, and whether the slice file is correct is checked, if yes, the slice receiving identifier of the receiving node database RecvData.sql is updated, the data transmission completion information is received, and then S34 is entered; if no, S32 is returned to continue transmitting the slice element information and slice data in a multi-thread parallel manner; S34, the receiving end combines and checks all the received data, if the data is correct, the data receiving identifier of the receiving node database RecvData.sql is updated; if no, the missing slice data is queried, the missing slice data information is transmitted to the collection end for data element information management, and then S32 is entered to transmit the slice element information and slice data in a multi-thread parallel manner.

7. The method of efficient data acquisition, transmission and processing in a complex closed environment according to claim 6, characterized in that, In S33, the receiving node database RecvData.sql is responsible for interacting with the receiving node and storing the received multi-source data information, and the receiving node database RecvData.sql includes the data table recv_file_info and recv_slice_info; The received file information is stored in the recv_file_info data table, and the received slice information is stored in the recv_slice_info data table; The recv_slice_info data table includes five fields of the received file name recv_file_name, the received slice number recv_slice_number, the received slice hash recv_slice_hash, the received slice size recv_slice_size and the slice receiving success identifier recv_slice_result; The recv_file_info data table includes five fields of a sending file name recv_file_name primary key, a sending file slice total recv_slice_total, a received file hash value recv_file_hash, a received file size recv_file_size, and a file receiving success identification recv_file_result.

8. The method of efficient data acquisition, transmission and processing in a complex closed environment according to claim 7, characterized in that, In S33, the receiving end processes the received slice data in parallel, and checks whether the slice file is correct. The specific process is as follows: The MD5 message digest algorithm is used to perform hash operation on the received slice data, and the hash value of the slice data is obtained. The calculated slice hash value is compared with the slice fingerprint information recv_slice_hash of the original data saved in the recv_slice_info table. If the two values are the same, it is indicated that the transmitted slice data is complete and correct. Otherwise, it is indicated that the slice data transmission is incorrect, and the slice data is retransmitted.

9. The method of claim 8, wherein the method comprises: In S34, the receiving end combines and checks all received data. The specific process is as follows: After all the slice data is received, the slice data is combined in the slice coding order, and the MD5 message digest algorithm is used to obtain the hash value of the whole data after the slice combination. The calculated hash value of the whole data is compared with the fingerprint information recv_file_hash of the original data saved in the recv_file_info table. If the two values are the same, it is indicated that the transmitted whole data is complete and correct, and the data receiving success identification recv_file_result in the recv_file_info table is updated to true. Otherwise, the missing slice data information in the database is queried, and real-time feedback is given to the collection end for data slice retransmission.

Images