A test data mass file block uploading and integrating processing method and system

Abstract

This invention discloses a method and system for uploading and integrating massive amounts of experimental data in chunks. The method involves generating a unique global identifier for each file on the client side, adaptively determining the chunk size based on the total file size, dividing the file into multiple chunks, and generating a unique chunk identifier for each chunk based on the file's unique global identifier, chunk number, and chunk data. The client initiates a resume upload query to the server, filters uploaded chunks, and then uses multi-threaded concurrent upload of unuploaded chunks. The server performs real-time verification based on the unique chunk identifiers, and after all chunks are uploaded, integrates them in parallel according to the chunk number to generate the target file. Integrity verification is completed by comparing the overall verification value with the file's unique global identifier. This invention achieves efficient resume upload of hundreds of gigabytes of experimental data, with an extremely low transmission error rate, reducing upload time by more than 60%, and is fully adaptable to various types of experimental data, exhibiting broad versatility and stability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and system for uploading and integrating massive amounts of experimental data in chunks. Background Technology

[0002] In high-end testing and research fields such as railways, aerospace, aviation, military industry and scientific research, the continuous iteration of testing operations has generated massive amounts of testing data. The capacity of a single test data file of various types, such as mechanics, environment, meteorology, and industrial testing, often reaches hundreds of GB, and the total amount of data in the overall business has even exceeded the TB level of storage scale.

[0003] With the surge in the volume of experimental data, traditional large file transfer and processing technologies are severely inadequate in adapting to multi-category, large-capacity experimental data, revealing many inherent defects in actual business implementation.

[0004] First, file upload and transmission efficiency is extremely low. Traditional solutions generally adopt a one-time upload mode for the entire file. For experimental data files of hundreds of gigabytes, the transmission process is highly susceptible to network fluctuations and link jitter. Once the transmission is interrupted, the existing mechanism does not have the ability to resume interrupted transmission and can only retransmit the entire file from the beginning. A single file upload often takes several hours, resulting in poor transmission fault tolerance and serious lack of business adaptability.

[0005] Secondly, there are potential conflicts between file storage and fragment verification. Existing conventional chunked upload technologies have not established a standardized globally unique file identification system, and the accuracy of independent fragment verification is insufficient. In practical applications, problems such as fragment loss, duplicate uploads, and fragment integration errors are common, with transmission integration error rates reaching 3% to 5%. This cannot meet the stringent requirements for the integrity, accuracy, and immutability of test data in critical fields such as aerospace and military industries.

[0006] Furthermore, the system suffers from high server resource consumption and significant bandwidth waste. Current technology fails to optimize fragmentation sizes and concurrent transmission strategies based on file size and data type differences, employing a single, fixed fragmentation rule for experimental files of varying specifications and types. When the server simultaneously handles multiple users uploading large files of different types, CPU utilization easily exceeds 80%, significantly increasing the risk of memory overflow. Simultaneously, the lack of an effective invalid fragmentation filtering mechanism results in redundant data transmission causing over 40% of network bandwidth to be unused, leading to low system concurrency capacity and resource utilization.

[0007] Meanwhile, existing similar solutions have significant technical limitations: general-purpose FTP and other chunked transfer tools are only suitable for ordinary office file transfers and are not customized for the specific business characteristics of experimental data. They cannot generate unique identifiers for multiple types of experimental files, and data distortion and file corruption are prone to occur after chunking and reassembly. They are also incompatible with the differentiated experimental data processing needs of mechanical time domain, frequency domain, and environmental monitoring. Existing data upload systems in the industrial field mostly adopt a rigid design with fixed chunk sizes (such as 5MB), which is not adapted to the transmission characteristics of hundreds of gigabytes of ultra-large files. They also do not build a precise location mechanism for resuming interrupted transmission based on file characteristics, resulting in low recovery efficiency after transmission interruption and insufficient compatibility with multiple types of experimental data. More importantly, existing mainstream solutions do not build a dual verification architecture of chunked MD5 and overall file MD5, but rely on a single verification method. They cannot ensure the accuracy of transmission of key experimental data throughout the entire process of transmission link, chunked storage, and whole-machine integration, and cannot meet the business management requirements of traceability, high security, and high reliability of high-end experimental data.

[0008] In summary, current traditional large file transfer and processing technologies and similar existing solutions have insurmountable technical shortcomings in terms of upload efficiency, integrity verification, resource scheduling, concurrency adaptation, and security traceability for hundreds of gigabytes of multi-type experimental data. They cannot meet the needs of efficient, secure, and reliable transmission and integration processing of massive experimental data in the fields of railway, aerospace, aviation, military, and scientific research. There is an urgent need to develop a new technology solution for the block-based upload and integration processing of massive large files that can adapt to multiple types of experimental data. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a method and system for uploading and integrating massive amounts of experimental data in chunks.

[0010] This invention discloses a method for uploading and integrating massive amounts of experimental data in chunks, including: Step S1: The client obtains the basic information of the various types of experimental data files to be uploaded. Based on the basic information and file content, a unique global identifier for the file is generated using the first encryption algorithm. The basic information includes the file name, total file size, and unique ID of the experimental task. Step S2: The client divides the multi-type test data file into multiple segments according to the total file size and the preset segmentation strategy, and generates a unique segment identifier for each segment. The unique segment identifier is generated based on the file's unique global identifier, the segment sequence number, and the segment data through a second encryption algorithm. Step S3: The client sends a breakpoint resume query request carrying the file's unique global identifier to the server, receives the uploaded fragment information returned by the server, and filters out the fragments that have not been uploaded. Step S4: The client uses a multi-threaded concurrency mechanism to upload the filtered unuploaded fragments and their corresponding fragment unique identifiers to the server. The server then performs real-time verification on each fragment based on the fragment unique identifier. Step S5: After the server detects that all fragments of a file have been uploaded, it performs parallel integration of all fragments according to the fragment sequence number to generate the target file, calculates the overall checksum of the target file, and compares it with the file's unique global identifier. If they match, the file storage is completed.

[0011] As a further improvement of the present invention, in step S1, the basic information further includes file type, creation time, and test type; the file type includes at least mechanical time domain data, environmental monitoring data, and radio frequency data.

[0012] As a further improvement of the present invention, in step S2, the multi-type test data file is divided into multiple fragments according to the total file size and a preset fragmentation strategy, specifically including: Multiple file size ranges are pre-defined, with different file size ranges corresponding to different fixed segment sizes; Based on the total size of the files to be uploaded, match the corresponding file size range, determine the corresponding segment size, and complete the file segmentation.

[0013] As a further improvement of the present invention, in step S2, each fragment obtained by division is encapsulated to form a standardized transmission unit; the standardized transmission unit carries at least the fragment sequence number, fragment unique identifier, file unique global identifier, fragment data, total number of fragments and file type information; the fragment unique identifier is generated by the second encryption algorithm.

[0014] As a further improvement of the present invention, step S3 specifically includes: The client sends a resume download query request to the server. The parameters of the query request include the file's unique global identifier, the total file size, the total number of fragments, and the file type. The server retrieves the file storage status based on the file's unique global identifier. If it is a complete file, it returns an "No upload required" flag; if it is a partial file, it returns a list of uploaded fragment numbers; if it is a new file, it returns an initial upload flag. The client receives the returned result and removes the uploaded fragment sequence number from the list of fragments to be uploaded, forming a transmission queue that only contains fragments that have not been uploaded.

[0015] As a further improvement of the present invention, in step S4, a multi-threaded concurrency mechanism is adopted, specifically including: the client obtains the current network bandwidth information, and according to the preset multiple bandwidth ranges and the fixed number of concurrent threads corresponding to each bandwidth range, matches the bandwidth range to which the current network bandwidth belongs, and dynamically determines the number of concurrent upload threads for this upload.

[0016] As a further improvement of the present invention, in step S5, all fragments are integrated in parallel according to the fragment sequence number, specifically including: The server arranges all uploaded fragments into an ordered sequence according to their fragment numbers, from smallest to largest. A multi-threaded parallel writing method is adopted to write the fragmented data in the ordered sequence to the target file storage path on the disk simultaneously or in batches, and a non-blocking I / O mechanism is used during the writing process.

[0017] As a further improvement of the present invention, in step S5, the overall verification value of the target file is calculated and compared with the file's unique global identifier, specifically including: The server uses the same first encryption algorithm as the client to calculate the overall verification value for the integrated target file. The overall verification value is compared with the unique global identifier of the file generated by the client in step S1. If the verification matches, the file integrity is deemed qualified. If the verification does not match, re-integration or re-upload is triggered.

[0018] As a further improvement of the present invention, the multi-type test data files include one or more of the following: mechanical test time-domain data files, mechanical test frequency-domain data files, environmental monitoring data files, meteorological data files, and aerospace test ballistic data files.

[0019] This invention discloses a system for uploading and integrating massive amounts of experimental data in chunks, which is used to execute the aforementioned method for uploading and integrating massive amounts of experimental data in chunks. The system includes a client module and a server module. The client module is used to collect basic information of various types of test data files, generate a unique global identifier for the file through a first encryption algorithm, complete intelligent file segmentation based on a preset segmentation strategy and generate a unique segment identifier, initiate a breakpoint resume query and filter out unuploaded segments, dynamically configure the number of concurrent threads according to network bandwidth and complete the concurrent uploading of segments. The server-side module is used to receive breakpoint resume query requests and return file upload status information, receive fragment data and perform real-time fragment verification based on the unique fragment identifier, and store fragment data after successful verification; it is also used to perform multi-threaded parallel integration to generate target files according to fragment sequence numbers after all fragments have been uploaded, complete the overall integrity verification of the file through the first encryption algorithm, and complete the file and metadata archiving after successful verification.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention combines basic file information and file content, employing first and second encryption algorithms respectively to generate a unique global file identifier and a unique fragment identifier, constructing a two-layer identity verification system. It also uses an adaptive fragmentation strategy based on file size ranges to split large files, and encapsulates the fragments into standardized transmission units. Relying on this dual encryption and verification mechanism, this invention can reduce the error rate of experimental data transmission to below 1 / 2^128, meeting the high accuracy and security requirements of experimental data. It can accurately distinguish between multi-task and multi-type experimental files, adapting to various business scenarios such as mechanics, environmental monitoring, and aerospace ballistics, standardizing fragmented transmission formats, and improving the uniqueness and security of large file uploads.

[0021] This invention relies on global file identification and fragment index management to achieve precise resume upload and automatic filtering of unuploaded fragments, achieving a 100% resume upload success rate. This effectively avoids duplicate transmissions and saves bandwidth resources. By dynamically adjusting concurrent threads through bandwidth range matching, it achieves intelligent fragmentation and concurrent scheduling, reducing upload time for 100GB-level files by more than 60%, while keeping server CPU utilization below 50%. It also supports simultaneous uploads of various types of 100GB-level experimental files by multiple users without bandwidth redundancy or waste. Combined with a real-time fragment verification mechanism, it can promptly identify data loss and tampering issues, significantly improving the stability of large file uploads and the system's versatility.

[0022] After all fragments have been uploaded, this invention sorts them by fragment number and uses multi-threaded parallel writing combined with a non-blocking I / O mechanism to complete file integration. Relying on the parallel integration algorithm, the integration time for hundreds of GB of files can be shortened to less than 15 minutes, breaking through the efficiency bottleneck of traditional single-threaded merging and reducing disk I / O blocking overhead. The integrated files are verified for integrity through a unified encryption algorithm. In case of an anomaly, re-integration or re-upload can be triggered, forming a closed-loop verification of the entire link, ensuring that the integrated test data files are complete and usable, and demonstrating significant engineering practical value. Attached Figure Description

[0023] Figure 1 This is a flowchart of a method for uploading and integrating massive amounts of experimental data in chunks, as disclosed in one embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings: like Figure 1 As shown, a method for uploading and integrating massive amounts of experimental data in chunks according to the present invention includes: Step S1: File Information Collection and Unique Identifier Generation The client obtains basic information about the various types of test data files to be uploaded. This basic information includes, but is not limited to, metadata such as file name, total file size, file type, creation time, and test type. The file types include at least mechanical time-domain data, environmental monitoring data, and radio frequency data.

[0026] The client generates a unique global identifier (MD5 value) for the file based on basic information and file content using a first encryption algorithm. Simultaneously, the client stores the file's MD5 value and related metadata for subsequent fragment verification and server-side interaction. This identifier generation method is compatible with all types of experimental data files without any differentiation restrictions. In a preferred embodiment, the MD5 encryption algorithm is used, with the file content as the core input, combined with the experimental task's unique ID, file name, and total file size as auxiliary input information for encryption calculation. Specifically, the MD5 algorithm processing in this embodiment includes: 1. Dividing the input information into 512-bit groups; 2. Further dividing each group into 16 32-bit sub-groups; 3. After a series of processing steps such as padding, initial vector initialization, and cyclic compression, four 32-bit groups are output; 4. Concatenating the four 32-bit groups to form a 128-bit hash value, i.e., the file's unique global identifier. This unique global identifier is used to uniquely represent the file throughout the entire upload, integration, and verification process and is compatible with all types of experimental data files.

[0027] Step S2: Intelligent Sharding and Sharding Identifier Generation The client divides various types of experimental data files into multiple fragments based on the total file size and a preset fragmentation strategy. Specifically, multiple file size ranges are pre-defined, each corresponding to a different fixed fragment size. The client matches the total file size of the file to be uploaded with the corresponding file size range to determine the appropriate fragment size and complete the file fragmentation. For example, when the total file size is ≤10GB, the fragment size is set to 10MB; when the total file size is between 10GB and 50GB, the fragment size is set to 20MB; when the total file size is between 50GB and 200GB, the fragment size is set to 50MB; and when the total file size is greater than 200GB, the fragment size is set to 100MB. The total number of fragments is calculated using the formula "Total number of fragments = ceil (total file bytes / fragment size)," ensuring that the last fragment size is not less than 1MB.

[0028] The client generates a unique identifier for each fragment. Specifically, each fragment is assigned a consecutive fragment number (incrementing from 0), and a second encryption algorithm (preferably, also using the MD5 algorithm) is used to generate a unique identifier for each fragment. The generation rule is "fragment MD5 value = MD5(file MD5 value + fragment number + fragment data)", ensuring the uniqueness and verifiability of the fragments.

[0029] Preferably, the client further encapsulates each of the divided fragments into a standardized transmission unit. This standardized transmission unit carries at least the fragment sequence number, fragment unique identifier, file unique global identifier, fragment data, total number of fragments, and file type information, facilitating differentiated identification and processing by the server.

[0030] Step S3: Resume interrupted download query and fragment filtering The client sends a resume request carrying the file's unique global identifier to the server. Specifically, the client sends the resume request to the server through the resume query interface (interface A). The request parameters include the file's unique global identifier (file MD5 value), the total file size, the total number of fragments, and the file type.

[0031] After receiving the request, the server retrieves the file storage status based on the file's unique global identifier: determining whether the file is a complete file (all fragments have been uploaded and integrated), a partial file (some fragments have been uploaded), or a new file (no fragments have been uploaded). The retrieval process does not distinguish between test file types. If the file has been fully uploaded and integrated, the "Complete File" flag is returned, and the client does not need to continue uploading; if the file has been partially uploaded, the list of uploaded fragment numbers is returned; if the file is a new file (no fragments have been uploaded), the initial upload flag is returned (e.g., fragment numbers start from 0).

[0032] After receiving the returned result, the client removes the uploaded fragment sequence number from the list of fragments to be uploaded, forming a transmission queue that only contains fragments that have not been uploaded, thereby avoiding duplicate uploads.

[0033] Step S4: Concurrent chunked upload and real-time verification The client uses a multi-threaded concurrency mechanism to upload the filtered unuploaded fragments and their corresponding fragment unique identifiers to the server. The server then performs real-time verification on each fragment based on the fragment unique identifier. Specifically, the client obtains the current network bandwidth information and, based on multiple preset bandwidth ranges and the fixed number of concurrent threads corresponding to each bandwidth range, matches the current network bandwidth to the specified bandwidth range and dynamically determines the number of concurrent upload threads for this upload. For example, when the network bandwidth is ≤100Mbps, 2 concurrent threads are used; when the network bandwidth is between 100Mbps and 1Gbps, 5 concurrent threads are used; and when the network bandwidth is greater than or equal to 1Gbps, 10 concurrent threads are used. The client supports user-configurable concurrent thread counts (range 1-10).

[0034] The client concurrently sends fragmented transmission requests to the server through the fragmented upload interface (interface B), with each request carrying the complete encapsulated data of a single fragment. Upon receiving a fragment, the server first verifies its integrity and correctness based on the fragment's unique identifier. If verification fails, a verification failure flag is returned, and the client re-uploads the fragment; if verification succeeds, the server stores the fragment data and records its upload status (completed / incomplete) and file type information.

[0035] Preferably, the client calculates the upload progress in real time (total size of uploaded segments / total file size) and displays the progress percentage and estimated remaining time to the user. This function is consistent across multiple types of experimental data files.

[0036] Step S5: Parallel integration of fragments and integrity verification Once the server detects that all fragments of a file have been uploaded, it integrates all fragments in parallel according to the fragment sequence number to generate the target file, calculates the overall checksum of the target file, and compares it with the file's unique global identifier. If they match, the file is stored in the database.

[0037] Specifically, the server arranges all uploaded fragments into an ordered sequence according to their fragment numbers from smallest to largest, ensuring that the integration order is consistent with the original files. The sorting rules do not distinguish between experimental file types. A multi-threaded parallel writing method is used to write the fragment data in the ordered sequence to the target file storage path on the disk simultaneously or in batches. Non-blocking I / O mechanisms can be used during the writing process to improve efficiency, and the corresponding storage format requirements are matched according to the file type.

[0038] After integration, the server uses the same first encryption algorithm (such as MD5) as the client to recalculate the overall checksum of the integrated target file and compares it with the unique global identifier of the file generated by the client in step S1. If the comparison matches, the file integrity is deemed acceptable, and the server stores the file information (including the file MD5 value, storage path, metadata, file type, test type, etc.) into the database through the file storage interface (interface C), completing the file storage and archiving, and returning an upload success identifier to the client. If the comparison does not match, a re-integration or re-upload process is triggered.

[0039] Preferably, the multiple types of test data files include one or more of the following: mechanical test time domain data files, mechanical test frequency domain data files, environmental monitoring data files, meteorological data files, and aerospace test ballistic data files.

[0040] Preferably, the present invention constructs a collaborative general architecture with three interfaces. Through the collaborative design of interface A (breakpoint resume query), interface B (fragmented upload), and interface C (file storage), the entire process of uploading, verification, integration, and storage is automated. The interface parameters include file type identifiers to adapt to the differentiated storage and management needs of various types of experimental data.

[0041] This invention also discloses a system for uploading and integrating massive amounts of experimental data in chunks, which is used to execute the above-mentioned method for uploading and integrating massive amounts of experimental data in chunks, and includes a client module and a server module: The client module is used to collect basic information of various types of experimental data files, generate a unique global identifier for the file through the first encryption algorithm, complete intelligent file segmentation based on the preset segmentation strategy and generate a unique segment identifier, initiate breakpoint resume query and filter unuploaded segments, dynamically configure the number of concurrent threads according to network bandwidth and complete the concurrent upload of segments. The server-side module is used to receive breakpoint resume query requests and return file upload status information, receive fragment data and perform real-time fragment verification based on the unique fragment identifier, and store fragment data after successful verification; it is also used to perform multi-threaded parallel integration to generate target files according to fragment sequence numbers after all fragments have been uploaded, complete the overall integrity verification of the file through the first encryption algorithm, and complete the file and metadata archiving after successful verification.

[0042] Test case Taking a 200GB time-domain data file from aerospace mechanics experiments and a 150GB meteorological data file from environmental monitoring as examples, the technical solution of this invention is used for uploading and integration: For a 200GB mechanical test data file, the MD5 value of the file was generated by combining the test task ID "ST-2024-058" and the file name "vibration_data_20240615.dat". An adaptive fragmentation strategy was adopted (fragment size 100MB, total fragments 2048). When uploading 1500 fragments, a network interruption was simulated. After reconnection, only the remaining 549 fragments were uploaded, and the success rate of resuming interrupted uploads was 100%. Ten threads were used for concurrent upload, which took 28 minutes. The parallel integration on the server side took 12 minutes. The MD5 verification was consistent after integration.

[0043] For a 150GB environmental monitoring data file, using the same process, with no interruptions, the concurrent upload of 8 threads took 22 minutes, and the parallel integration on the server side took 9 minutes. The MD5 checksums were consistent after integration.

[0044] Both types of experimental data files were uploaded and integrated efficiently and accurately, verifying the universality and adaptability of the technical solution of this invention to multiple types of experimental data.

[0045] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for processing test data mass file block upload integration, characterized in that, include: Step S1: The client obtains the basic information of the various types of experimental data files to be uploaded. Based on the basic information and file content, a unique global identifier for the file is generated using the first encryption algorithm. The basic information includes the file name, total file size, and unique ID of the experimental task. Step S2: The client divides the multi-type test data file into multiple segments according to the total file size and the preset segmentation strategy, and generates a unique segment identifier for each segment. The unique segment identifier is generated based on the file's unique global identifier, the segment sequence number, and the segment data through a second encryption algorithm. Step S3: The client sends a breakpoint resume query request carrying the file's unique global identifier to the server, receives the uploaded fragment information returned by the server, and filters out the fragments that have not been uploaded. Step S4: The client uses a multi-threaded concurrency mechanism to upload the filtered unuploaded fragments and their corresponding fragment unique identifiers to the server. The server then performs real-time verification on each fragment based on the fragment unique identifier. Step S5: After the server detects that all fragments of a file have been uploaded, it performs parallel integration of all fragments according to the fragment sequence number to generate the target file, calculates the overall checksum of the target file, and compares it with the file's unique global identifier. If they match, the file storage is completed.

2. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, In step S1, the basic information also includes file type, creation time, and test type; the file type includes at least mechanical time-domain data, environmental monitoring data, and radio frequency data.

3. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, In step S2, the multi-type experimental data file is divided into multiple fragments according to the total file size and a preset fragmentation strategy, specifically including: Multiple file size ranges are pre-defined, with different file size ranges corresponding to different fixed segment sizes; Based on the total size of the files to be uploaded, match the corresponding file size range, determine the corresponding segment size, and complete the file segmentation.

4. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, In step S2, each fragment obtained from the division is encapsulated to form a standardized transmission unit; the standardized transmission unit carries at least the fragment sequence number, fragment unique identifier, file unique global identifier, fragment data, total number of fragments, and file type information; the fragment unique identifier is generated by the second encryption algorithm.

5. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, Step S3 specifically includes: The client sends a resume download query request to the server. The parameters of the query request include the file's unique global identifier, the total file size, the total number of fragments, and the file type. The server retrieves the file storage status based on the file's unique global identifier. If it is a complete file, it returns an "No upload required" flag; if it is a partial file, it returns a list of uploaded fragment numbers; if it is a new file, it returns an initial upload flag. The client receives the returned result and removes the uploaded fragment sequence number from the list of fragments to be uploaded, forming a transmission queue that only contains fragments that have not been uploaded.

6. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, In step S4, a multi-threaded concurrency mechanism is adopted, specifically including: the client obtains the current network bandwidth information, matches the bandwidth range to which the current network bandwidth belongs based on multiple preset bandwidth ranges and the fixed number of concurrent threads corresponding to each bandwidth range, and dynamically determines the number of concurrent upload threads for this upload.

7. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, In step S5, all fragments are integrated in parallel according to their fragment number, specifically including: The server arranges all uploaded fragments into an ordered sequence according to their fragment numbers, from smallest to largest. A multi-threaded parallel writing method is adopted to write the fragmented data in the ordered sequence to the target file storage path on the disk simultaneously or in batches, and a non-blocking I / O mechanism is used during the writing process.

8. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, In step S5, the overall checksum of the target file is calculated and compared with the file's unique global identifier, specifically including: The server uses the same first encryption algorithm as the client to calculate the overall verification value for the integrated target file. The overall verification value is compared with the unique global identifier of the file generated by the client in step S1. If the verification matches, the file integrity is deemed qualified. If the verification does not match, re-integration or re-upload is triggered.

9. The method for processing massive amounts of experimental data by uploading and integrating data in chunks according to claim 1, characterized in that, The various types of test data files include one or more of the following: mechanical test time-domain data files, mechanical test frequency-domain data files, environmental monitoring data files, meteorological data files, and aerospace test ballistic data files.

10. A system for uploading and integrating massive amounts of experimental data in chunks, used to execute the method for uploading and integrating massive amounts of experimental data in chunks as described in any one of claims 1-9, characterized in that... Includes client-side and server-side modules: The client module is used to collect basic information of various types of test data files, generate a unique global identifier for the file through a first encryption algorithm, complete intelligent file segmentation based on a preset segmentation strategy and generate a unique segment identifier, initiate a breakpoint resume query and filter out unuploaded segments, dynamically configure the number of concurrent threads according to network bandwidth and complete the concurrent uploading of segments. The server-side module is used to receive breakpoint resume query requests and return file upload status information, receive fragment data and perform real-time fragment verification based on the unique fragment identifier, and store fragment data after successful verification; it is also used to perform multi-threaded parallel integration to generate target files according to fragment sequence numbers after all fragments have been uploaded, complete the overall integrity verification of the file through the first encryption algorithm, and complete the file and metadata archiving after successful verification.

Images