A campus data collection method and device, a storage medium and an equipment

Abstract

The application provides a campus data collection method, device, storage medium and equipment; wherein the method is: a client generates file catalog information according to to-be-uploaded attachment file data; a server writes the file catalog information into a to-be-written sequence; a preprocessing sequence is formed; the server divides the preprocessing sequence into multiple subfiles and stores the subfiles into a database; the client divides the to-be-uploaded attachment file data into multiple subfiles and generates an index table, performs merging and dividing operations on each subfile of the client and generates an index table of the to-be-uploaded attachment file data; and the client sequentially uploads the index table and each subfile corresponding to the index table to the database for storage. The method can reduce the transmission data size between the client and the server, reduce the transmission resource occupation, optimize the transmission efficiency, avoid long-time occupation of the server storage space and network channel bandwidth, and improve the data collection speed.

Description

Technical Field

[0001] The present invention relates to the field of data acquisition technology, and more specifically, to a campus data acquisition method, device, storage medium and equipment. Background Art

[0002] When collecting data on campus, we often rely on importing and exporting data from various office systems. While much of the content is similar, much of it is heterogeneous data from different systems. After manually completing the data on the office system forms, various attachments must be uploaded before the process can be completed. However, because the attachment systems underlying office systems are all the same, the first type limits attachment files to a fixed size, while the second type has no restrictions on file size or format. The first method is very troublesome if the file size and format are different when uploading, or if the file size exceeds the system limit. File processing, format conversion, or compression to reduce the file size before uploading is very complicated. The second method adds a burden to the office system's storage and management. If someone maliciously uploads multiple large files, server storage space and network channel bandwidth will be occupied, making it impossible for others to upload attachments. Summary of the Invention

[0003] In order to overcome the shortcomings and deficiencies in the prior art, the purpose of the present invention is to provide a campus data collection method, device, storage medium and equipment; this method can reduce the size of transmitted data between the client and the server, reduce the transmission resource occupation, optimize the transmission efficiency, will not occupy the server storage space and network channel bandwidth for a long time, and improve the speed of data collection.

[0004] In order to achieve the above object, the present invention is implemented by the following technical solution: a campus data collection method, characterized in that it includes the following steps:

[0005] Step 1: The client generates file description information based on the attachment file data to be uploaded, and sends the file description information to the server; the file description information includes the file header, file name, file number and file data size of the attachment file data;

[0006] Step 2: The server writes the file description information uploaded by the client into the waiting-to-be-written sequence, using the uploaded file description information as the current file description information; the waiting-to-be-written sequence is a queue consisting of the file description information uploaded by each client;

[0007] Step 3: Compare the header of the current file description information with the headers of the other file description information in the sequence to be written, and extract the file description information in the sequence to be written that has the same header as the current file description information, except for the current file description information, to form a preprocessing sequence;

[0008] Step 4: The server divides the attachment file data corresponding to each file description information in the preprocessing sequence into multiple subfiles of a size less than or equal to a threshold value N1, and obtains an index table for each attachment file data and each subfile corresponding to the index table; the server stores the index table for each attachment file data and each subfile corresponding to the index table in a database; the index table for the attachment file data is a sequential table consisting of the file header, file name, file number, and file data size of all the subfiles after the attachment is divided;

[0009] Step 5: The client splits the attachment file data to be uploaded into multiple sub-files of a size less than or equal to the threshold N1 and generates an index table of the attachment file data to be uploaded and each sub-file corresponding to the index table. The client's sub-files are sequentially compared with the sub-files in the database, and the client's sub-files are merged and split to generate an index table of the attachment file data to be uploaded.

[0010] In step 6, the client uploads the index table of the attachment file data to be uploaded and each sub-file corresponding to the index table to the database for storage.

[0011] Preferably, in step 5, the method of sequentially comparing each sub-file of the client with the sub-files in the database of the server, performing merging and splitting operations on each sub-file of the client and generating an index table of the attachment file data to be uploaded is:

[0012] Step 5.1, let the sub-file set formed by each sub-file of the client be G = {g i}; i∈[1,M1], M1 is the total number of client sub-files; g i is the i-th subfile in G; set the variable j, j∈[1,M2], M2 is the total number of subfiles divided by the attachment file data corresponding to each file catalog information in the server's preprocessing sequence, and set the initial value of i and j to 1;

[0013] Step 5.2, calculate g i The hash value of the j-th sub-file in the database is calculated as the first hash value; the hash value of the j-th sub-file in the database is calculated as the second hash value;

[0014] Step 5.3: Determine whether the first hash value matches the second hash value:

[0015] If it matches, go to step 5.3.1: add the index table of the attachment file data to be uploaded to g i Delete the file header, file name, file number, and file data size of the j-th sub-file in the database, and write the file header, file name, file number, file data, and the position of the sub-file in the database into the index table of the attachment file data to be uploaded; then jump to step 5.4;

[0016] If there is no match, then execute step 5.3.2: search the database for the hash value with the largest cosine similarity between the hash value of the sub-file and the first hash value, and mark the sub-file corresponding to the hash value as the reference file; or search the database for the sub-file corresponding to the hash value with the largest file access value between the hash value of the sub-file and the first hash value, and mark it as the reference file;

[0017] Determine the size of N3 and N2, where N2 is the size of the smallest subfile among all subfiles in the subfile set G, and N3 is the size of the reference file: if N3>N2, then split the reference file into a first subfile and a second subfile, with the size of the first subfile being N2 and the size of the second subfile being N3-N2; otherwise, use the reference file as the first subfile and the size of the second subfile being 0;

[0018] Determine the size of N4 and N2, where N4 is the i-th sub-file g i Size: If N4>N2, then the i-th sub-file g i Split into the third and fourth sub-files, the size of the third sub-file is N2, the size of the fourth sub-file is N4-N2; otherwise, the i-th sub-file g i As the third sub-file, the size of the fourth sub-file is 0;

[0019] Calculate a hash value H1 of the first sub-file, a hash value H2 of the second sub-file, a hash value H3 of the third sub-file, and a hash value H4 of the fourth sub-file;

[0020] Determine the size of hash values ​​H1 and H3: If H1 and H3 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the fourth subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the fourth subfile and the first subfile in the database are written into the index table of the attachment file data to be uploaded, and the fourth subfile is added to the subfile set G;

[0021] Otherwise, determine the size of the hash values ​​H2 and H4: If H2 and H4 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded iThe file header, file name, file number and file data size of the third subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the third subfile and the second subfile in the database are written into the index table of the attachment file data to be uploaded, and the third subfile is added to the subfile set G;

[0022] Step 5.4: If j ≤ M2, increase the value of j by 1 and go to step 5.2; if j > M2, set the value of j to 1 and go to step 5.5;

[0023] Step 5.5: if i≤M1, increase the value of i by 1 and go to step 5.2; if i>M1, obtain the index table of the attachment file data to be uploaded, and use the sub-file set G as each sub-file of the attachment file data to be uploaded.

[0024] Preferably, in step 5.3.2, the file access value AC between the hash value of the x-th sub-file in the database and the first hash value is x The calculation method is:

[0025]

[0026] Among them, y is the cumulative variable, y∈[1,M1]; k x,y is the cosine similarity between the hash value of the x-th subfile in the database and the hash value of the y-th subfile on the client; MaxK is the maximum value of the cosine similarity between the hash values ​​of each subfile in the database and the first hash value, and log is the logarithmic function; MeanK is the arithmetic mean of the cosine similarities between the hash values ​​of each subfile in the database and the first hash value.

[0027] Preferably, step 7 is further included after step 6: when the client reads the attachment file data in the database, the server merges the files corresponding to the index table in the database in sequence according to the index table of the attachment file data to obtain the attachment file data.

[0028] Preferably, in step 1, the attachment file data includes text files, pictures, compressed packages, and video files.

[0029] Preferably, in step 1, the file number is a hash value of the attachment file data to be uploaded.

[0030] Preferably, in step 4 and step 5, the value range of the threshold N1 is [5, 20] MB.

[0031] A campus data collection device, characterized by comprising a client, a server and a database;

[0032] The client is configured to generate and send file catalog information according to the to-be-uploaded attachment file data;

[0033] The server is configured to write the file catalog information uploaded by the client into a to-be-written sequence, and take the uploaded file catalog information as current file catalog information; the to-be-written sequence is a queue formed by the file catalog information uploaded by each client;

[0034] The server is further configured to sequentially judge the file header of the current file catalog information with the file headers of the remaining file catalog information in the to-be-written sequence, extract each file catalog information in the to-be-written sequence other than the current file catalog information and having the same file header as the current file catalog information, and form a pre-processing sequence;

[0035] The server is further configured to divide the attachment file data corresponding to each file catalog information in the pre-processing sequence into a plurality of sub-files with a size less than or equal to a threshold N1, obtain an index table of each attachment file data and each sub-file corresponding to the index table, and sequentially upload the index table of each attachment file data and each sub-file corresponding to the index table.

[0036] The database is configured to store the index table of each attachment file data and each sub-file corresponding to the index table.

[0037] The client is further configured to divide the to-be-uploaded attachment file data into a plurality of sub-files with a size less than or equal to a threshold N1, generate an index table of the to-be-uploaded attachment file data and each sub-file corresponding to the index table, sequentially compare each sub-file of the client with the sub-files in the database, perform a merging and dividing operation on each sub-file of the client, and generate an index table of the to-be-uploaded attachment file data; and sequentially upload the index table of the to-be-uploaded attachment file data and each sub-file corresponding to the index table.

[0038] The database is further configured to receive and store the index table of the attachment file data and each sub-file corresponding to the index table.

[0039] When the client reads the attachment file data in the database, the server sequentially performs file merging on each file corresponding to the index table in the database according to the index table of the attachment file data, thereby obtaining the attachment file data.

[0040] A storage medium, characterized in that: the storage medium stores a computer program, and the computer program causes a processor to execute the above-mentioned campus data collection method when the computer program is executed by the processor.

[0041] A computing device, comprising a processor and a memory for storing a processor-executable program, characterized in that: the processor executes the program stored in the memory to implement the above-mentioned campus data collection method.

[0042] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0043] The present invention divides files into sub-files, fully utilizing similar sub-files stored in the server and database to reduce the size of data transmitted between the client and the server, thereby reducing the transmission resource occupation; there is no need to convert the file format or compress the file to reduce the file size before uploading, thus optimizing the transmission efficiency, not occupying the server storage space and network channel bandwidth for a long time, and improving the speed of data acquisition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a campus data collection method of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] like Figure 1 As shown, this embodiment provides a campus data collection method, comprising the following steps:

[0048] Step 1: The client generates file description information based on the attachment file data to be uploaded, and sends the file description information to the server; the file description information includes the file header, file name, file number and file data size of the attachment file data; the attachment file data includes text files, pictures, compressed packages, and video files; the file number is the hash value of the attachment file data to be uploaded.

[0049] Step 2: The server writes the file description information uploaded by the client into the waiting-to-be-written sequence, using the uploaded file description information as the current file description information; the waiting-to-be-written sequence is a queue consisting of the file description information uploaded by each client.

[0050] Step 3: Compare the file header of the current file description information with the file headers of the remaining file description information in the sequence to be written in turn, and extract the file description information in the sequence to be written that is the same as the file header of the current file description information except the current file description information to form a preprocessing sequence.

[0051] In step 4, the server divides the attachment file data corresponding to each file description in the preprocessing sequence into multiple subfiles with a size less than or equal to a threshold value N1. The value range of threshold value N1 is preferably [5, 20] MB. The server obtains an index table for each attachment file data and the subfiles corresponding to the index table. The server stores the index table for each attachment file data and the subfiles corresponding to the index table in a database. The index table for the attachment file data is a sequential table consisting of the file header, file name, file number, and file data size of all the subfiles after the attachment is divided. The index table in step 4 is the index table corresponding to each file in the preprocessing sequence. Since multiple clients may upload attachment file data simultaneously, each attachment file data must correspond to an index table.

[0052] In step 5, the client divides the attachment file data to be uploaded into multiple sub-files with a size less than or equal to the threshold N1 and generates an index table of the attachment file data to be uploaded and the sub-files corresponding to the index table, and compares the sub-files of the client with the sub-files in the database in turn, merges and divides the sub-files of the client and generates an index table of the attachment file data to be uploaded.

[0053] Specifically, in step 5, each sub-file of the client is sequentially compared with the sub-files in the server's database, and each sub-file of the client is merged and split to generate an index table of the attachment file data to be uploaded.

[0054] Step 5.1, let the sub-file set formed by each sub-file of the client be G = {g i}; i∈[1,M1], M1 is the total number of client sub-files; g i is the i-th subfile in G; set the variable j, j∈[1,M2], M2 is the total number of subfiles divided by the attachment file data corresponding to each file catalog information in the server's preprocessing sequence, and set the initial value of i and j to 1;

[0055] Step 5.2, calculate g i The hash value of the j-th sub-file in the database is calculated as the first hash value; the hash value of the j-th sub-file in the database is calculated as the second hash value;

[0056] Step 5.3: Determine whether the first hash value matches the second hash value:

[0057] If it matches, go to step 5.3.1: add the index table of the attachment file data to be uploaded to g i Delete the file header, file name, file number, and file data size of the j-th sub-file in the database, and write the file header, file name, file number, file data, and the position of the sub-file in the database into the index table of the attachment file data to be uploaded; then jump to step 5.4;

[0058] If there is no match, then execute step 5.3.2: search the database for the hash value with the largest cosine similarity between the hash value of the sub-file and the first hash value, and mark the sub-file corresponding to the hash value as the reference file; or search the database for the sub-file corresponding to the hash value with the largest file access value between the hash value of the sub-file and the first hash value, and mark it as the reference file;

[0059] The file access value AC between the hash value of the x-th sub-file in the database and the first hash value x The calculation method is as follows, where x is the sequence number of the sub-file in the database:

[0060]

[0061] Among them, y is the cumulative variable, y∈[1,M1]; k x,y is the cosine similarity between the hash value of the x-th subfile in the database and the hash value of the y-th subfile on the client; MaxK is the maximum value of the cosine similarity between the hash values ​​of each subfile in the database and the first hash value, where log is the logarithmic function; MeanK is the arithmetic mean of the cosine similarities between the hash values ​​of each subfile in the database and the first hash value;

[0062] Preferably, the hash value in the present invention refers to a file hash value.

[0063] Preferably, the method for calculating the cosine similarity between hash values ​​is: converting the hash value into a string and calculating the cosine similarity between each string. The calculation of the hash value in the present invention can also be replaced by calculating the Hamming distance, and the cosine similarity in each step can be replaced by the Hamming distance.

[0064] Determine the size of N3 and N2, where N2 is the size of the smallest subfile among all subfiles in the subfile set G, and N3 is the size of the reference file: if N3>N2, then split the reference file into a first subfile and a second subfile, with the size of the first subfile being N2 and the size of the second subfile being N3-N2; otherwise, use the reference file as the first subfile and the size of the second subfile being 0;

[0065] Determine the size of N4 and N2, where N4 is the i-th sub-file g i Size: If N4>N2, then the i-th sub-file g i Split into the third and fourth sub-files, the size of the third sub-file is N2, the size of the fourth sub-file is N4-N2; otherwise, the i-th sub-file g i As the third sub-file, the size of the fourth sub-file is 0;

[0066] Calculate a hash value H1 of the first sub-file, a hash value H2 of the second sub-file, a hash value H3 of the third sub-file, and a hash value H4 of the fourth sub-file;

[0067] Determine the size of hash values ​​H1 and H3: If H1 and H3 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the fourth subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the fourth subfile and the first subfile in the database are written into the index table of the attachment file data to be uploaded, and the fourth subfile is added to the subfile set G;

[0068] Otherwise, determine the size of the hash values ​​H2 and H4: If H2 and H4 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the third subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the third subfile and the second subfile in the database are written into the index table of the attachment file data to be uploaded, and the third subfile is added to the subfile set G;

[0069] Step 5.4: If j ≤ M2, increase the value of j by 1 and go to step 5.2; if j > M2, set the value of j to 1 and go to step 5.5;

[0070] Step 5.5: if i≤M1, increase the value of i by 1 and go to step 5.2; if i>M1, obtain the index table of the attachment file data to be uploaded, and use the sub-file set G as each sub-file of the attachment file data to be uploaded.

[0071] In step 6, the client uploads the index table of the attachment file data to be uploaded and each sub-file corresponding to the index table to the database for storage.

[0072] Step 7: When the client reads the attachment file data in the database, the server merges the files corresponding to the index table in the database in sequence according to the index table of the attachment file data to obtain the attachment file data.

[0073] The present invention divides files into sub-files, fully utilizing similar sub-files stored in the server and database to reduce the size of data transmitted between the client and the server, thereby reducing the transmission resource occupation; there is no need to convert the file format or compress the file to reduce the file size before uploading, thus optimizing the transmission efficiency, not occupying the server storage space and network channel bandwidth for a long time, and improving the speed of data acquisition.

[0074] To implement the above campus data collection method, this embodiment provides a campus data collection device, including a client, a server and a database;

[0075] The client is used to generate file description information based on the data of the attachment file to be uploaded and send it;

[0076] The server is used to write the file description information uploaded by the client into the waiting-to-write sequence, using the uploaded file description information as the current file description information; the waiting-to-write sequence is a queue composed of the file description information uploaded by each client;

[0077] The server is further configured to sequentially compare the header of the current file description information with the headers of the remaining file description information in the sequence to be written, and extract the file description information in the sequence to be written that has the same header as the current file description information, except for the current file description information, to form a pre-processing sequence;

[0078] The server is further configured to divide the attachment file data corresponding to each file description information in the preprocessing sequence into a plurality of subfiles of a size less than or equal to a threshold value N1, and obtain an index table of each attachment file data and each subfile corresponding to the index table;

[0079] The database is used to store the index table of each attachment file data and each sub-file corresponding to the index table;

[0080] The client is further configured to divide the attachment file data to be uploaded into a plurality of sub-files of a size less than or equal to a threshold value N1 and generate an index table of the attachment file data to be uploaded and each sub-file corresponding to the index table, and sequentially compare each sub-file of the client with the sub-files in the database, perform a merging and dividing operation on each sub-file of the client and generate an index table of the attachment file data to be uploaded; and sequentially upload the index table of the attachment file data to be uploaded and each sub-file corresponding to the index table;

[0081] The database is also used to receive and store the index table of the attachment file data and each sub-file corresponding to the index table;

[0082] When the client reads the attachment file data in the database, the server sequentially performs file merging on each file corresponding to the index table in the database according to the index table of the attachment file data, thereby obtaining the attachment file data.

[0083] Embodiment two

[0084] The embodiment is a storage medium, wherein the storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to perform the campus data collection method in the embodiment one.

[0085] Embodiment three

[0086] The embodiment is a computing device, comprising a processor and a memory for storing a program executable by the processor, and the processor, when executing the program stored in the memory, implements the campus data collection method in the embodiment one.

[0087] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and all are included in the protection scope of the present application.

Claims

1. A campus data collection method, characterized by: The following steps are involved: Step 1: The client generates file description information based on the attachment file data to be uploaded, and sends the file description information to the server; the file description information includes the file header, file name, file number and file data size of the attachment file data; Step 2: The server writes the file description information uploaded by the client into the waiting-to-be-written sequence, using the uploaded file description information as the current file description information; the waiting-to-be-written sequence is a queue consisting of the file description information uploaded by each client; Step 3: Compare the header of the current file description information with the headers of the other file description information in the sequence to be written, and extract the file description information in the sequence to be written that has the same header as the current file description information, except for the current file description information, to form a preprocessing sequence; Step 4: The server divides the attachment file data corresponding to each file description information in the preprocessing sequence into multiple sub-files with a size less than or equal to a threshold value N1, and obtains an index table for each attachment file data and each sub-file corresponding to the index table; The server stores the index table of each attachment file data and each sub-file corresponding to the index table in the database; The index table of the attachment file data is a sequence table consisting of the file header, file name, file number, and file data size of all sub-files after the attachment is split; Step 5: The client splits the attachment file data to be uploaded into multiple sub-files of a size less than or equal to the threshold N1 and generates an index table of the attachment file data to be uploaded and each sub-file corresponding to the index table. The client's sub-files are sequentially compared with the sub-files in the database, and the client's sub-files are merged and split to generate an index table of the attachment file data to be uploaded. Step 6: The client uploads the index table of the attachment file data to be uploaded and each sub-file corresponding to the index table to the database for storage; Step 7: When the client reads the attachment file data in the database, the server merges the files corresponding to the index table in the database in sequence according to the index table of the attachment file data to obtain the attachment file data; In step 5, the method of sequentially comparing each sub-file of the client with the sub-files in the server database, performing merging and splitting operations on each sub-file of the client and generating an index table of the attachment file data to be uploaded is: Step 5.1, let the sub-file set formed by each sub-file of the client be G = {g i }; i∈[1,M1], M1 is the total number of client sub-files; g i is the i-th subfile in G; Set the variable j, j∈[1,M2], where M2 is the total number of sub-files of the attachment file data corresponding to each file description information in the server's pre-processing sequence, and set the initial values ​​of i and j to 1; Step 5.2, calculate g i The hash value of the j-th sub-file in the database is calculated as the first hash value; the hash value of the j-th sub-file in the database is calculated as the second hash value; Step 5.3: Determine whether the first hash value matches the second hash value: If it matches, go to step 5.3.1: add the index table of the attachment file data to be uploaded to g i Delete the file header, file name, file number, and file data size of the j-th sub-file in the database, and write the file header, file name, file number, file data, and the position of the sub-file in the database into the index table of the attachment file data to be uploaded; then jump to step 5.4; If there is no match, then execute step 5.3.2: search the database for the hash value with the largest cosine similarity between the hash value of the sub-file and the first hash value, and mark the sub-file corresponding to the hash value as the reference file; or search the database for the sub-file corresponding to the hash value with the largest file access value between the hash value of the sub-file and the first hash value, and mark it as the reference file; Determine the size of N3 and N2, where N2 is the size of the smallest subfile among all subfiles in the subfile set G, and N3 is the size of the reference file: if N3>N2, then split the reference file into a first subfile and a second subfile, with the size of the first subfile being N2 and the size of the second subfile being N3-N2; otherwise, use the reference file as the first subfile and the size of the second subfile being 0; Determine the size of N4 and N2, where N4 is the i-th sub-file g i Size: If N4>N2, then the i-th sub-file g i Split into the third and fourth sub-files, the size of the third sub-file is N2, the size of the fourth sub-file is N4-N2; otherwise, the i-th sub-file g i As the third sub-file, the size of the fourth sub-file is 0; Calculate a hash value H1 of the first sub-file, a hash value H2 of the second sub-file, a hash value H3 of the third sub-file, and a hash value H4 of the fourth sub-file; Determine the size of hash values ​​H1 and H3: If H1 and H3 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the fourth subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the fourth subfile and the first subfile in the database are written into the index table of the attachment file data to be uploaded, and the fourth subfile is added to the subfile set G; Otherwise, determine the size of the hash values ​​H2 and H4: If H2 and H4 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the third subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the third subfile and the second subfile in the database are written into the index table of the attachment file data to be uploaded, and the third subfile is added to the subfile set G; Step 5.4: If j ≤ M2, increase the value of j by 1 and go to step 5.2; if j > M2, set the value of j to 1 and go to step 5.5; Step 5.5: if i≤M1, increase the value of i by 1 and go to step 5.2; if i>M1, obtain the index table of the attachment file data to be uploaded, and use the sub-file set G as each sub-file of the attachment file data to be uploaded.

2. The campus data collection method according to claim 1, characterized in that: In step 5.3.2, the file access value AC between the hash value of the x-th sub-file in the database and the first hash value x The calculation method is: Among them, y is the cumulative variable, y∈[1,M1]; k x,y is the cosine similarity between the hash value of the x-th subfile in the database and the hash value of the y-th subfile on the client; MaxK is the maximum value of the cosine similarity between the hash values ​​of each subfile in the database and the first hash value, and log is the logarithmic function; MeanK is the arithmetic mean of the cosine similarities between the hash values ​​of each subfile in the database and the first hash value.

3. The campus data collection method according to claim 1, wherein: After step 6, the method further includes step 7: when the client reads the attachment file data in the database, the server sequentially merges the files corresponding to the index table in the database according to the index table of the attachment file data to obtain the attachment file data.

4. The campus data collection method according to claim 1, wherein: In step 1, the attachment file data includes text files, pictures, compressed packages, and video files.

5. The campus data collection method according to claim 1, characterized in that: In step 1, the file number is the hash value of the attachment file data to be uploaded.

6. The campus data collection method according to claim 1, characterized in that: In step 4 and step 5, the value range of the threshold N1 is [5, 20] MB.

7. A campus data collection device, characterized by: Including client, server and database; The client is used to generate file description information based on the data of the attachment file to be uploaded and send it; The server is used to write the file description information uploaded by the client into the waiting-to-write sequence, using the uploaded file description information as the current file description information; the waiting-to-write sequence is a queue composed of the file description information uploaded by each client; The server is further configured to sequentially compare the header of the current file description information with the headers of the remaining file description information in the sequence to be written, and extract the file description information in the sequence to be written that has the same header as the current file description information, except for the current file description information, to form a pre-processing sequence; The server is further configured to divide the attachment file data corresponding to each file description information in the preprocessing sequence into a plurality of sub-files of a size less than or equal to a threshold value N1, and obtain an index table of each attachment file data and each sub-file corresponding to the index table; The index table of each attachment file data and each sub-file corresponding to the index table; The database is used to store the index table of each attachment file data and each sub-file corresponding to the index table; The client is further configured to split the attachment file data to be uploaded into a plurality of sub-files of a size less than or equal to a threshold value N1 and generate an index table of the attachment file data to be uploaded and each sub-file corresponding to the index table, and sequentially compare each sub-file of the client with the sub-files in the database, perform a merging and splitting operation on each sub-file of the client and generate an index table of the attachment file data to be uploaded; Used to upload the index table of the attachment file data to be uploaded and each sub-file corresponding to the index table in sequence; The database is also used to receive and store the index table of the attachment file data and each sub-file corresponding to the index table; The method of sequentially comparing each sub-file of the client with the sub-files in the database, performing merging and splitting operations on each sub-file of the client and generating an index table of the attachment file data to be uploaded is as follows: Step 5.1, let the sub-file set formed by each sub-file of the client be G = {g i }; i∈[1,M1], M1 is the total number of client sub-files; g i is the i-th subfile in G; set the variable j, j∈[1,M2], M2 is the total number of subfiles divided by the attachment file data corresponding to each file catalog information in the server's preprocessing sequence, and set the initial value of i and j to 1; Step 5.2, calculate g i The hash value of the j-th sub-file in the database is calculated as the first hash value; the hash value of the j-th sub-file in the database is calculated as the second hash value; Step 5.3: Determine whether the first hash value matches the second hash value: If it matches, go to step 5.3.1: add the index table of the attachment file data to be uploaded to g i Delete the file header, file name, file number, and file data size of the j-th sub-file in the database, and write the file header, file name, file number, file data, and the position of the sub-file in the database into the index table of the attachment file data to be uploaded; then jump to step 5.4; If there is no match, then execute step 5.3.2: search the database for the hash value with the largest cosine similarity between the hash value of the sub-file and the first hash value, and mark the sub-file corresponding to the hash value as the reference file; or search the database for the sub-file corresponding to the hash value with the largest file access value between the hash value of the sub-file and the first hash value, and mark it as the reference file; Determine the size of N3 and N2, where N2 is the size of the smallest subfile among all subfiles in the subfile set G, and N3 is the size of the reference file: if N3>N2, then split the reference file into a first subfile and a second subfile, with the size of the first subfile being N2 and the size of the second subfile being N3-N2; otherwise, use the reference file as the first subfile and the size of the second subfile being 0; Determine the size of N4 and N2, where N4 is the i-th sub-file g i Size: If N4>N2, then the i-th sub-file g i Split into the third and fourth sub-files, the size of the third sub-file is N2, the size of the fourth sub-file is N4-N2; otherwise, the i-th sub-file g i As the third sub-file, the size of the fourth sub-file is 0; Calculate a hash value H1 of the first sub-file, a hash value H2 of the second sub-file, a hash value H3 of the third sub-file, and a hash value H4 of the fourth sub-file; Determine the size of hash values ​​H1 and H3: If H1 and H3 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the fourth subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the fourth subfile and the first subfile in the database are written into the index table of the attachment file data to be uploaded, and the fourth subfile is added to the subfile set G; Otherwise, determine the size of the hash values ​​H2 and H4: If H2 and H4 are equal, then the i-th sub-file g i Delete it from the sub-file set G and add the i-th sub-file g in the index table of the attachment file data to be uploaded i The file header, file name, file number and file data size of the third subfile are deleted, and the reference file is deleted from the database, the first subfile and the second subfile are stored in the database, the file header, file name, file number, file data and the position of the subfile in the database of the third subfile and the second subfile in the database are written into the index table of the attachment file data to be uploaded, and the third subfile is added to the subfile set G; Step 5.4: If j ≤ M2, increase the value of j by 1 and go to step 5.2; if j > M2, set the value of j to 1 and go to step 5.5; Step 5.5: if i≤M1, increase the value of i by 1 and go to step 5.2; if i>M1, obtain the index table of the attachment file data to be uploaded, and use the sub-file set G as each sub-file of the attachment file data to be uploaded.

8. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the campus data collection method according to any one of claims 1 to 6.

9. A computing device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the campus data collection method according to any one of claims 1 to 6 is implemented.

Images