Distributed file upload verification method, device, equipment and storage medium

Abstract

The present invention relates to a distributed file upload verification method. The method comprises the following steps: obtaining a file to be uploaded, splitting the file to be uploaded into multiple data blocks according to a preset threshold, and uploading the multiple data blocks to a distributed file system; obtaining upload information of each data block through the distributed file system, and storing the upload information of each data block in an Uploading linked list; integrating the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and uploading the integrated data to an Upload-Result linked list; monitoring the integrated data in the Upload-Result linked list in real time, and when detecting the presence of a successfully uploaded data block, comparing the successfully uploaded data block through the distributed file system; if an exception occurs in a successfully uploaded data block, notifying a user that the upload is abnormal; and if all multiple data blocks are normal, notifying the user that the upload is successful. The present invention ensures the integrity and accuracy of data by monitoring and verifying the upload status of each data block in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file storage systems, and in particular to a distributed file upload verification method, device, equipment and storage medium. Background Art

[0002] In the modern information society, data volumes are increasing dramatically, especially driven by big data and cloud computing technologies. This has led to a significant increase in the demand for uploading various types of files. Distributed file storage systems (DFS) have become an essential tool for managing and storing large amounts of data. However, existing distributed file storage systems still face many technical challenges and issues in the file upload process.

[0003] In distributed file storage systems, file uploads are susceptible to network conditions. For example, factors such as network latency, bandwidth limitations, and congestion can lead to unstable data transfer speeds and even data loss or corruption. These issues are particularly pronounced when uploading large files, potentially causing extended upload times or even upload failures.

[0004] Due to the uncertainty of network transmission, file uploads may experience "packet loss"—that is, some data blocks are lost or damaged during transmission, resulting in an incomplete file. This not only affects the user experience but can also lead to the loss or damage of important data, and in serious cases, may even cause data security issues.

[0005] Most existing distributed file storage systems lack effective real-time monitoring and feedback mechanisms, making it difficult to monitor data integrity during file uploads. Users often only know whether a file has been successfully uploaded and is complete after the upload is complete. This delayed feedback mechanism fails to meet users' demands for real-time performance and reliability.

[0006] Due to the potential for data loss or corruption during the upload process, existing systems often re-upload entire files or large blocks of data, increasing data redundancy and storage costs. Furthermore, the lack of intelligent decision-making mechanisms prevents effective differentiation between data blocks requiring re-upload and those that have already been successfully uploaded, further increasing storage and transmission costs. Summary of the Invention

[0007] The main purpose of the present invention is to provide a distributed file upload verification method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology is difficult to ensure data integrity during the file upload process.

[0008] To achieve the above object, the present invention provides a distributed file upload verification method, which includes the following steps:

[0009] Obtain a file to be uploaded, split the file to be uploaded into multiple data blocks according to a preset threshold, upload the multiple data blocks to a distributed file system, obtain upload information of each data block through the distributed file system, and store the upload information of each data block in an Uploading linked list;

[0010] Integrate the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and upload the integrated data to the Upload-Result linked list;

[0011] monitoring the integrated data of the Upload-Result linked list in real time, and when detecting a successfully uploaded data block, comparing the successfully uploaded data block with the distributed file system;

[0012] If the successfully uploaded data block has an abnormality, the user is prompted that the upload is abnormal; if all of the multiple data blocks have no abnormalities, the user is prompted that the upload is successful.

[0013] Optionally, after the step of if the successfully uploaded data block has an abnormality, the method further includes:

[0014] The abnormal data blocks are deleted from the distributed file system.

[0015] Optionally, after the step of real-time monitoring of the integrated data of the Upload-Result linked table, the method further includes:

[0016] When it is detected that there is a data block that fails to be uploaded, the data block that fails to be uploaded is marked and the data block that fails to be uploaded is deleted from the distributed file system.

[0017] Optionally, after the step of prompting the user of the upload abnormality if the successfully uploaded data block has an abnormality; and prompting the user of the upload success if none of the multiple data blocks have an abnormality, the method further includes:

[0018] Integrate all successfully uploaded data block information into the Monitor-Result linked list. When the user re-uploads the file to be uploaded, determine the data blocks to be uploaded that need to be re-uploaded according to the Monitor-Result linked list, and upload the data blocks to be uploaded to the distributed file system;

[0019] If the data block to be uploaded is uploaded successfully, the user is prompted that the upload is successful.

[0020] Optionally, after the step of integrating the information of all successfully uploaded data blocks into the Monitor-Result linked list, the method further includes:

[0021] Delete the upload information corresponding to all successfully uploaded data block information from the Upload-Result linked list.

[0022] Optionally, after the step of successfully uploading the data block to be uploaded, the method further includes:

[0023] All successfully uploaded data block information in the Monitor-Result linked list is persistently stored in the database. If the persistent storage is successful, the persistently stored data block information in the Monitor-Result linked list is deleted.

[0024] Optionally, the step of uploading the multiple data blocks to a distributed file system includes:

[0025] The multiple data blocks are uploaded to the distributed file system in a multi-threaded uploading manner.

[0026] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a distributed file upload verification device, which includes:

[0027] A file splitting and uploading module obtains a file to be uploaded, splits the file to be uploaded into multiple data blocks according to a preset threshold, uploads the multiple data blocks to a distributed file system, obtains upload information of each data block through the distributed file system, and stores the upload information of each data block in an Uploading linked list;

[0028] An upload information integration module integrates the upload information in the Uploading list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and uploads the integrated data to the Upload-Result list;

[0029] An upload status monitoring module monitors the integrated data of the Upload-Result linked list in real time, and when a successfully uploaded data block is detected, compares the successfully uploaded data block with the distributed file system;

[0030] The verification result prompt module prompts the user that the upload is abnormal if there is any abnormality in the successfully uploaded data block; and prompts the user that the upload is successful if there is no abnormality in all the multiple data blocks.

[0031] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a distributed file upload verification device, which includes a memory, a processor, and a distributed file upload verification program stored in the memory and executable on the processor. When the distributed file upload verification program is executed by the processor, the steps of the distributed file upload verification method described above are implemented.

[0032] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a distributed file upload verification program is stored. When the distributed file upload verification program is executed by a processor, the steps of the distributed file upload verification method described above are implemented.

[0033] The present invention relates to a distributed file upload verification method. The method comprises the following steps: obtaining a file to be uploaded, splitting the file to be uploaded into multiple data blocks according to a preset threshold, and uploading the multiple data blocks to a distributed file system; obtaining upload information of each data block through the distributed file system, and storing the upload information of each data block in an Uploading linked list; integrating the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and uploading the integrated data to an Upload-Result linked list; monitoring the integrated data in the Upload-Result linked list in real time, and when detecting the existence of a successfully uploaded data block, comparing the successfully uploaded data block through the distributed file system; if an exception exists in the successfully uploaded data block, prompting a user that the upload is abnormal; and prompting a user that the upload is successful if all the multiple data blocks are normal. The present invention ensures the integrity and accuracy of the data by monitoring and verifying the upload status of each data block in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the distributed file upload verification device of the present invention;

[0035] Figure 2 This is a flow chart of a first embodiment of a distributed file upload verification method according to the present invention;

[0036] Figure 3 This is a functional module diagram of a preferred embodiment of the distributed file upload verification device of the present invention.

[0037] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] It's important to note that in the modern information society, data volumes are increasing dramatically, especially driven by big data and cloud computing technologies. This has led to a significant increase in the demand for uploading various types of files. Distributed file storage systems (DFS) have become an essential tool for managing and storing large amounts of data. However, existing distributed file storage systems still face many technical challenges and issues in the file upload process.

[0040] In distributed file storage systems, file uploads are susceptible to network conditions. For example, factors such as network latency, bandwidth limitations, and congestion can lead to unstable data transfer speeds and even data loss or corruption. These issues are particularly pronounced when uploading large files, potentially causing extended upload times or even upload failures.

[0041] Due to the uncertainty of network transmission, file uploads may experience "packet loss"—that is, some data blocks are lost or damaged during transmission, resulting in an incomplete file. This not only affects the user experience but can also lead to the loss or damage of important data, and in serious cases, may even cause data security issues.

[0042] Most existing distributed file storage systems lack effective real-time monitoring and feedback mechanisms, making it difficult to monitor data integrity during file uploads. Users often only know whether a file has been successfully uploaded and is complete after the upload is complete. This delayed feedback mechanism fails to meet users' demands for real-time performance and reliability.

[0043] Due to the potential for data loss or corruption during the upload process, existing systems often re-upload entire files or large blocks of data, increasing data redundancy and storage costs. Furthermore, the lack of intelligent decision-making mechanisms prevents effective differentiation between data blocks requiring re-upload and those that have already been successfully uploaded, further increasing storage and transmission costs.

[0044] Based on the above defects, the present invention provides a distributed file upload verification device, referring to Figure 1 , Figure 1 This is a structural diagram of the device hardware operating environment involved in the embodiment of the distributed file upload verification device of the present invention.

[0045] like Figure 1As shown, the distributed file upload verification device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0046] Those skilled in the art will understand that Figure 1 The hardware structure of the distributed file upload verification device shown in the figure does not constitute a limitation on the distributed file upload verification device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0047] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a distributed file upload verification program. The operating system is a program that manages and controls the distributed file upload verification device and software resources, supporting the operation of the network communication module, the user interface module, the distributed file upload verification program, and other programs or software. The network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.

[0048] exist Figure 1 In the hardware structure of the distributed file upload verification device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the distributed file upload verification program stored in the memory 1005 and perform the following operations:

[0049] Obtain a file to be uploaded, split the file to be uploaded into multiple data blocks according to a preset threshold, upload the multiple data blocks to a distributed file system, obtain upload information of each data block through the distributed file system, and store the upload information of each data block in an Uploading linked list;

[0050] Integrate the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and upload the integrated data to the Upload-Result linked list;

[0051] monitoring the integrated data of the Upload-Result linked list in real time, and when detecting a successfully uploaded data block, comparing the successfully uploaded data block with the distributed file system;

[0052] If the successfully uploaded data block has an abnormality, the user is prompted that the upload is abnormal; if all of the multiple data blocks have no abnormalities, the user is prompted that the upload is successful.

[0053] Furthermore, after the step of if the successfully uploaded data block has an abnormality, the method further includes:

[0054] The abnormal data blocks are deleted from the distributed file system.

[0055] Furthermore, after the step of real-time monitoring of the integrated data of the Upload-Result linked table, the method further includes:

[0056] When it is detected that there is a data block that has failed to be uploaded, the data block that has failed to be uploaded is marked and the data block that has failed to be uploaded is deleted from the distributed file system.

[0057] Furthermore, after the step of prompting the user that the upload is abnormal if the successfully uploaded data block has an abnormality, and prompting the user that the upload is successful if none of the multiple data blocks have an abnormality, the method further includes:

[0058] Integrate all successfully uploaded data block information into the Monitor-Result linked list. When the user re-uploads the file to be uploaded, determine the data blocks to be uploaded that need to be re-uploaded according to the Monitor-Result linked list, and upload the data blocks to be uploaded to the distributed file system;

[0059] If the data block to be uploaded is uploaded successfully, the user is prompted that the upload is successful.

[0060] Furthermore, after the step of integrating the information of all successfully uploaded data blocks into the Monitor-Result linked list, the method further includes:

[0061] Delete the upload information corresponding to all successfully uploaded data block information from the Upload-Result linked list.

[0062] Furthermore, after the step of successfully uploading the data block to be uploaded, the method further includes:

[0063] All successfully uploaded data block information in the Monitor-Result linked list is persistently stored in the database. If the persistent storage is successful, the persistently stored data block information in the Monitor-Result linked list is deleted.

[0064] Furthermore, the step of uploading the plurality of data blocks to the distributed file system includes:

[0065] The multiple data blocks are uploaded to the distributed file system in a multi-threaded uploading manner.

[0066] The specific implementation of the distributed file upload verification device of the present invention is basically the same as the various embodiments of the distributed file upload verification method described below, and will not be repeated here.

[0067] The present invention also provides a distributed file upload verification method based on the above-mentioned distributed file upload verification device.

[0068] Reference Figure 2 , Figure 2 This is a flow chart of the first embodiment of the distributed file upload verification method of the present invention.

[0069] The embodiment of the present invention provides an embodiment of a distributed file upload verification method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0070] In various embodiments of the distributed file upload verification method, the execution entity is a distributed file upload verification device.

[0071] The distributed file upload verification method is applied to a distributed file upload verification device, and the distributed file upload verification method includes:

[0072] Step S10: Obtain a file to be uploaded, split the file to be uploaded into multiple data blocks according to a preset threshold, upload the multiple data blocks to a distributed file system, obtain upload information of each data block through the distributed file system, and store the upload information of each data block in an Uploading linked list;

[0073] In the modern information society, data volumes are increasing dramatically, especially driven by big data and cloud computing technologies. This has led to a significant increase in the demand for uploading various types of files. Distributed file storage systems (DFS) have become an essential tool for managing and storing large amounts of data. However, existing distributed file storage systems still face many technical challenges and issues in the file upload process.

[0074] In distributed file storage systems, file uploads are susceptible to network conditions. For example, factors such as network latency, bandwidth limitations, and congestion can lead to unstable data transfer speeds and even data loss or corruption. These issues are particularly pronounced when uploading large files, potentially causing extended upload times or even upload failures.

[0075] Due to the uncertainty of network transmission, file uploads may experience "packet loss"—that is, some data blocks are lost or damaged during transmission, resulting in an incomplete file. This not only affects the user experience but can also lead to the loss or damage of important data, and in serious cases, may even cause data security issues.

[0076] Most existing distributed file storage systems lack effective real-time monitoring and feedback mechanisms, making it difficult to monitor data integrity during file uploads. Users often only know whether a file has been successfully uploaded and is complete after the upload is complete. This delayed feedback mechanism fails to meet users' demands for real-time performance and reliability.

[0077] Due to the potential for data loss or corruption during the upload process, existing systems often re-upload entire files or large blocks of data, increasing data redundancy and storage costs. Furthermore, the lack of intelligent decision-making mechanisms prevents effective differentiation between data blocks requiring re-upload and those that have already been successfully uploaded, further increasing storage and transmission costs.

[0078] In this embodiment, the user selects a file to be uploaded through the client. The file can be any type of file, such as a document, a picture, a video, etc. Inside the system, the file to be uploaded is loaded into the memory and prepared for processing.

[0079] The file to be uploaded is split into several data blocks based on a preset threshold. The preset threshold can be set based on factors such as network bandwidth and file size. Assuming the preset threshold is 20MB, a 100MB file will be split into 5 data blocks.

[0080] The split data blocks are uploaded to the Distributed File System (DFS) in parallel using multiple threads. Multi-threaded uploads can fully utilize network bandwidth and system resources, improving upload efficiency. Each thread is responsible for uploading one or more data blocks.

[0081] After each data block is uploaded to DFS, the system will return the upload status information of the data block, including whether the upload is successful, storage location, data block size, etc. This information is used for subsequent verification and management.

[0082] The upload information of each data block is stored in a temporary linked list (Uploading linked list) for unified management and subsequent processing. The Uploading linked list records the upload status information of all data blocks.

[0083] Through the above steps, the system can effectively split large files and upload them to the distributed file system, while ensuring that the upload status and information of each data block are accurately recorded and managed, ensuring data integrity and reliability during the file upload process.

[0084] Example description:

[0085] Get the file to upload: The user selects a 100MB file named large_file.zip on the client. The system loads large_file.zip into memory and prepares it for processing.

[0086] Split file: The preset threshold is 20MB. The system splits large_file.zip into 5 data blocks according to the 20MB threshold:

[0087] Data block 1: large_file_part1.zip, size 20MB

[0088] Data block 2: large_file_part2.zip, size 20MB

[0089] Data block 3: large_file_part3.zip, size 20MB

[0090] Data block 4: large_file_part4.zip, size 20MB

[0091] Data block 5: large_file_part5.zip, size 20MB

[0092] Multi-threaded data block upload: The system starts five threads, each responsible for uploading a data block to DFS. Thread 1 uploads large_file_part1.zip, thread 2 uploads large_file_part2.zip, and so on. After each thread completes the upload, the system obtains the upload status information for that data block from DFS.

[0093] Get upload information: For example, if large_file_part1.zip is uploaded successfully, the returned information includes:

[0094] Upload Status: Success

[0095] Storage location: dfs: / / path / to / large_file_part1.zip

[0096] Data block size: 20MB

[0097] Storing upload information in the Uploading list: The system stores the upload information of each data block in the Uploading list:

[0098] Uploading list:

[0099] -Data block 1: upload status (success), storage location (dfs: / / path / to / large_file_part1.zip), size (20MB);

[0100] -Data block 2: upload status (success), storage location (dfs: / / path / to / large_file_part2.zip), size (20MB);

[0101] -Data block 3: upload status (success), storage location (dfs: / / path / to / large_file_part3.zip), size (20MB);

[0102] -Data block 4: upload status (success), storage location (dfs: / / path / to / large_file_part4.zip), size (20MB);

[0103] -Data block 5: upload status (success), storage location (dfs: / / path / to / large_file_part5.zip), size (20MB).

[0104] Step S20: Integrate the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and upload the integrated data to the Upload-Result linked list;

[0105] In this embodiment, the Uploading linked list stores the upload information of each data block, including upload status, storage location, data block size, etc. The system first obtains the upload information from the Uploading linked list.

[0106] The upload information in the Uploading list is integrated to generate an integrated data structure containing all data block upload information. The integrated data contains detailed upload information for each data block, such as upload status, storage location, data block size, etc.

[0107] After the integrated data is generated, the system uploads it to the Upload-Result linked list. The Upload-Result linked list is used to store the complete uploaded information of all data blocks for subsequent real-time monitoring and verification.

[0108] Through the above steps, the uploaded information of all data blocks can be effectively integrated and managed to ensure the data integrity and reliability during the file upload process, while providing comprehensive data support for subsequent monitoring and verification.

[0109] Example description:

[0110] Obtaining upload information from the Uploading list: The system reads the upload information for each data block from the Uploading list. Assume that the Uploading list records that the file "large_file.zip" is split into 5 data blocks, and the upload information for each data block is as follows:

[0111] Data block 1: upload status (success), storage location (path in the distributed file system), size (20MB);

[0112] Data block 2: upload status (success), storage location (path in the distributed file system), size (20MB);

[0113] Data block 3: upload status (success), storage location (path in the distributed file system), size (20MB);

[0114] Data block 4: upload status (success), storage location (path in the distributed file system), size (20MB);

[0115] Data block 5: upload status (success), storage location (path in the distributed file system), size (20MB).

[0116] Integrate the uploaded information: The system integrates the uploaded information in the Uploading list and generates an integrated data structure containing detailed information of all data blocks:

[0117] {

[0118] "File name":"large_file.zip",

[0119] "Data block information":[

[0120] {"Data Block":1,"Upload Status":"Success","Storage Location":"dfs: / / path / to / large_file_part1.zip","Size":"20MB"},

[0121] {"Data Block":2,"Upload Status":"Success","Storage Location":"dfs: / / path / to / large_file_part2.zip","Size":"20MB"},

[0122] {"Data Block":3,"Upload Status":"Success","Storage Location":"dfs: / / path / to / large_file_part3.zip","Size":"20MB"},

[0123] {"Data Block":4,"Upload Status":"Success","Storage Location":"dfs: / / path / to / large_file_part4.zip","Size":"20MB"},

[0124] {"Data Block":5,"Upload Status":"Success","Storage Location":"dfs: / / path / to / large_file_part5.zip","Size":"20MB"} ]

[0126] }

[0127] Upload the integrated data to the Upload-Result linked list.

[0128] Step S30: monitor the integrated data of the Upload-Result linked list in real time, and when a successfully uploaded data block is detected, compare the successfully uploaded data block with the distributed file system;

[0129] In this embodiment, the system continuously monitors the aggregated data in the Upload-Result linked list to ensure that the upload status of each data block is promptly checked and processed. For example, the Upload-Result linked list records the upload information for each data block of the file "large_file.zip." The system continuously monitors the linked list, checking the upload status of each data block, paying particular attention to successfully uploaded data blocks.

[0130] Assume that the system detects that there are successfully uploaded data blocks in the Upload-Result linked list, such as data block 1, data block 2, and data block 3. The system marks these data blocks as "pending verification" and records detailed information such as their storage location and size. For example:

[0131] Data block 1: Status (pending verification), storage location (path in the distributed file system), size (20MB);

[0132] Data block 2: Status (pending verification), storage location (path in the distributed file system), size (20MB);

[0133] Data block 3: Status (pending verification), storage location (path in the distributed file system), size (20MB).

[0134] The system reads the data block information marked as "to be verified" from the Upload-Result linked list and compares it through the distributed file system.

[0135] The comparison operation includes: verifying the integrity of the data block to ensure that the data block has not been modified or damaged; verifying the consistency of the data block to ensure that the uploaded data block is consistent with the original data block.

[0136] After the comparison is successful, the system updates the status of the data block to "Verification Successful". If any abnormality is found, the user will be prompted and appropriate processing will be performed.

[0137] Through the above steps, each successfully uploaded data block can be effectively monitored and verified, ensuring the data integrity and reliability during the file upload process, while providing comprehensive data support for subsequent exception handling.

[0138] Example description:

[0139] Real-time monitoring of the Upload-Result linked table: The system monitors the integrated data in the Upload-Result linked table in real time. Assume that the Upload-Result linked table records the upload information of each data block of the file "large_file.zip".

[0140] Detecting successfully uploaded data blocks: The system detects the presence of successfully uploaded data blocks in the Upload-Result linked list, such as data blocks 1, 2, and 3. The system marks these blocks as "pending verification" and records detailed information such as their storage location and size.

[0141] Comparison via the distributed file system: The system reads data blocks marked "pending verification" from the Upload-Result linked list and compares them via the distributed file system. This comparison verifies the integrity and consistency of the data blocks, ensuring that the uploaded data blocks are identical to the original data blocks. Upon successful comparison, the system updates the data block status to "Verification Successful." If any anomalies are detected, the user is notified and appropriate action is taken.

[0142] Step S40: If the successfully uploaded data block has an abnormality, the user is prompted that the upload is abnormal; if all the multiple data blocks have no abnormality, the user is prompted that the upload is successful.

[0143] In this embodiment, the system checks the integrity and consistency of each successfully uploaded data block during the comparison process. For example, the file "large_file.zip" is split into 5 data blocks, and the system compares these data blocks one by one.

[0144] Suppose the system compares data blocks 1, 2, and 3 and confirms that none of them are abnormal. When comparing data block 4, the system detects an anomaly, such as an incomplete data block or inconsistency with the original data. The system records the anomaly and prompts the user to upload the anomaly. For example, the prompt might read: "Data block 4 upload error. Please re-upload this data block."

[0145] When comparing data block 5, the system confirms that there is no anomaly in the data block. Because the system detected an anomaly in data block 4 during the comparison process, the system will not prompt the upload success, but wait for the user to process the abnormal data block.

[0146] If the system confirms that all successfully uploaded data blocks are normal (for example, after the user re-uploads data block 4 and confirms that there are no abnormalities), the system will prompt the user that the upload was successful. The prompt message may be: "The file large_file.zip has been successfully uploaded, and all data blocks are normal."

[0147] Through the above steps, each successfully uploaded data block can be effectively processed and verified, ensuring the data integrity and reliability during the file upload process, and prompting the user to take timely measures when an exception occurs, ultimately ensuring that the file is successfully uploaded and the data is complete.

[0148] Example description:

[0149] Checking the successfully uploaded data blocks: During the comparison process, the system checks the five data blocks of the file "large_file.zip". During the comparison process, the system confirms that data blocks 1, 2, and 3 are normal.

[0150] Handling abnormal data blocks: When comparing data block 4, the system detects an abnormality in the data block (such as incomplete data or inconsistency with the original data). The system records the abnormality and prompts the user: "Data block 4 upload abnormality, please re-upload the data block."

[0151] Confirm that all data blocks are normal: When comparing data block 5, the system confirms that there are no abnormalities in this data block. Since there is an abnormality in data block 4, the system will not prompt that the upload is successful, but will wait for the user to process the abnormal data block.

[0152] The user is prompted that the upload was successful: After the user re-uploads data block 4, the system re-checks and confirms that there are no anomalies. At this time, the system prompts the user: "The file large_file.zip has been successfully uploaded, and all data blocks are normal."

[0153] The present invention relates to a distributed file upload verification method. The method comprises the following steps: obtaining a file to be uploaded, splitting the file to be uploaded into multiple data blocks according to a preset threshold, and uploading the multiple data blocks to a distributed file system; obtaining upload information of each data block through the distributed file system, and storing the upload information of each data block in an Uploading linked list; integrating the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and uploading the integrated data to an Upload-Result linked list; monitoring the integrated data in the Upload-Result linked list in real time, and when detecting the existence of a successfully uploaded data block, comparing the successfully uploaded data block through the distributed file system; if an exception exists in the successfully uploaded data block, prompting a user that the upload is abnormal; and prompting a user that the upload is successful if all the multiple data blocks are normal. The present invention ensures the integrity and accuracy of the data by monitoring and verifying the upload status of each data block in real time.

[0154] Furthermore, a second embodiment of the distributed file upload verification method of the present invention is proposed. In the above step S40, after the step of checking if the successfully uploaded data block has an abnormality, the method further includes:

[0155] Step a1: Deleting abnormal data blocks from the distributed file system.

[0156] In this embodiment, when comparing data blocks, the system checks the integrity and consistency of each successfully uploaded data block. If an anomaly is found in the data block (for example, the data block is incomplete or inconsistent with the original data), the system will record the anomaly.

[0157] When an abnormality is detected in a successfully uploaded data block, the system will delete the abnormal data block from the distributed file system to ensure the accuracy and integrity of the data.

[0158] The system records the exception and prompts the user to upload the exception, informing the user that the data block with the exception needs to be uploaded again.

[0159] Example description:

[0160] Checking the successfully uploaded data blocks: During the comparison process, the system checks the five data blocks of the file "large_file.zip". During the comparison process, the system confirms that data blocks 1, 2, and 3 are normal.

[0161] Processing abnormal data blocks: When comparing data block 4, the system detects an abnormality in the data block (such as incomplete data or inconsistency with the original data). The system records the abnormality and decides to delete data block 4.

[0162] Deleting the abnormal data block: The system deletes the storage record of data block 4 through the distributed file system API or management interface. This operation ensures that data block 4 no longer exists in the distributed file system.

[0163] Prompt the user about upload exceptions: The system records the exception and prompts the user: "Data block 4 has been uploaded abnormally and has been deleted from the distributed file system. Please upload the data block again."

[0164] Through the above steps, this embodiment can effectively process and delete abnormal data blocks, ensure the accuracy and integrity of data in the distributed file system, and promptly prompt the user to re-upload abnormal data blocks to ensure that the file is finally uploaded completely.

[0165] Furthermore, a third embodiment of the distributed file upload verification method of the present invention is proposed. In the above step S30, after the step of real-time monitoring of the integrated data of the Upload-Result linked table, the method further includes:

[0166] Step b1: When a data block that fails to be uploaded is detected, the data block that fails to be uploaded is marked and the data block that fails to be uploaded is deleted from the distributed file system.

[0167] In this embodiment, the system monitors the aggregated data in the Upload-Result linked list in real time. For example, the Upload-Result linked list records the upload information for each data block of the file "large_file.zip." The system continuously monitors the linked list, checking the upload status of each data block, paying particular attention to any data blocks that failed to upload.

[0168] Assume that the system detects data blocks in the Upload-Result linked list that failed to be uploaded, such as data blocks 4 and 5. The system marks these blocks as "upload failed" and records detailed information such as their storage location and size. For example:

[0169] Data block 4: status (upload failed), storage location (path in the distributed file system), size (20MB);

[0170] Data block 5: Status (upload failed), storage location (path in the distributed file system), size (20MB).

[0171] The system deletes the data blocks marked as "failed to upload" through the distributed file system's API or management interface. The system also deletes the storage records for data blocks 4 and 5 to ensure that these failed upload blocks no longer exist in the distributed file system.

[0172] Example description:

[0173] Real-time monitoring of the integrated data in the Upload-Result linked table: The system monitors the integrated data in the Upload-Result linked table in real time. Assume that the Upload-Result linked table records the upload information of each data block of the file "large_file.zip".

[0174] Marking data blocks that failed to upload: The system detects data blocks in the Upload-Result list that failed to upload, such as data blocks 4 and 5. The system marks these blocks as "failed to upload" and records detailed information such as their storage location and size.

[0175] Deleting data blocks that failed to upload: The system deletes data blocks marked as "failed to upload" through the distributed file system's API or management interface. The system deletes the storage records for data blocks 4 and 5 to ensure that these failed upload data blocks no longer exist in the distributed file system.

[0176] Through the above steps, this embodiment can effectively monitor and process data blocks that fail to be uploaded, ensuring the accuracy and integrity of data in the distributed file system while avoiding data redundancy and storage space waste.

[0177] Furthermore, a fourth embodiment of the distributed file upload verification method of the present invention is proposed, which, after the above step S40, further includes:

[0178] Step c1: Integrate all successfully uploaded data block information into the Monitor-Result linked list. When the user re-uploads the file to be uploaded, determine the data blocks to be uploaded that need to be re-uploaded according to the Monitor-Result linked list, and upload the data blocks to be uploaded to the distributed file system.

[0179] Step c2: If the data block to be uploaded is uploaded successfully, the user is prompted that the upload is successful.

[0180] In this embodiment, the system checks the integrity and consistency of each successfully uploaded data block to confirm whether there are any anomalies. If there are any anomalies, the user is prompted to upload the anomaly; if all data blocks are normal, the user is prompted to upload successfully.

[0181] After confirming that all successfully uploaded data blocks are normal, the system integrates the information of these data blocks into the Monitor-Result linked table. The Monitor-Result linked table is used to record the information of all successfully uploaded data blocks for subsequent monitoring and processing.

[0182] When the user re-uploads the file to be uploaded, the system determines the data blocks to be uploaded that need to be re-uploaded according to the Monitor-Result linked list, and uploads these data blocks to the distributed file system.

[0183] If the data block to be uploaded is successfully re-uploaded, the system will prompt the user that the upload is successful.

[0184] Example description:

[0185] Detecting and processing successfully uploaded data blocks: The system compares the five data blocks of the file "large_file.zip" and finds an anomaly in data block 4. The user is prompted to re-upload the data block.

[0186] Integrate successful data block information into the Monitor-Result linked list: The system confirms that there are no exceptions in data block 1, data block 2, data block 3, and data block 5, and integrates the information of these data blocks into the Monitor-Result linked list.

[0187] The user re-uploads the abnormal data block: When the user re-uploads the file "large_file.zip", the system determines the data block 4 that needs to be re-uploaded based on the Monitor-Result linked list.

[0188] Confirm the re-uploaded data block: The system confirms that data block 4 was successfully re-uploaded and there are no abnormalities. The system adds the information of data block 4 to the Monitor-Result linked list and prompts the user that the upload was successful.

[0189] Through the above steps, this embodiment can effectively process and verify each successfully uploaded data block, ensure the data integrity and reliability during the file upload process, and promptly prompt the user to take measures when an exception occurs, ultimately ensuring that the file is successfully uploaded and the data is complete.

[0190] Furthermore, a fifth embodiment of the distributed file upload verification method of the present invention is proposed. In the above step c1, after the step of integrating the information of all successfully uploaded data blocks into the Monitor-Result linked list, the method further includes:

[0191] Step d1: Delete the upload information corresponding to all successfully uploaded data block information from the Upload-Result linked list.

[0192] In this embodiment, the system integrates all successfully uploaded data block information from the Upload-Result linked list into the Monitor-Result linked list, ensuring that the successful data block information is uniformly recorded and managed.

[0193] After successfully integrating the data block information into the Monitor-Result linked list, the system will delete the corresponding uploaded information from the Upload-Result linked list to prevent data redundancy and confusion, ensuring that only the data block information that needs further processing is retained in the linked list.

[0194] Example description:

[0195] The system confirms that data blocks 1, 2, 3, and 5 of the file "large_file.zip" have been uploaded successfully and without any exceptions. The system integrates the information of these data blocks from the Upload-Result linked list into the Monitor-Result linked list. For example:

[0196] Monitor-Result linked list:

[0197] - File name: large_file.zip

[0198] -Data block 1: status (success), storage location (dfs: / / path / to / large_file_part1.zip), size (20MB);

[0199] -Data block 2: status (success), storage location (dfs: / / path / to / large_file_part2.zip), size (20MB);

[0200] -Data block 3: status (success), storage location (dfs: / / path / to / large_file_part3.zip), size (20MB);

[0201] -Data block 5: Status (success), storage location (dfs: / / path / to / large_file_part5.zip), size (20MB).

[0202] Delete successful data block information from the Upload-Result chain: After the system integrates the successful data block information into the Monitor-Result chain, it deletes the successful data block information from the Upload-Result chain. For example:

[0203] Upload-Result linked list (updated):

[0204] - File name: large_file.zip;

[0205] -Data block 4: status (failed), storage location (dfs: / / path / to / large_file_part4.zip), size (20MB).

[0206] Through the above steps, this embodiment can effectively integrate and manage the successfully uploaded data block information, and at the same time delete the successfully uploaded information in the Upload-Result linked table, thereby ensuring the accuracy and conciseness of data records.

[0207] Furthermore, a sixth embodiment of the distributed file upload verification method of the present invention is proposed. In the above step c2, after the step of verifying that the data block to be uploaded is successfully uploaded, the method further includes:

[0208] In step e1, all successfully uploaded data block information in the Monitor-Result linked list is persistently stored in the database. If the persistent storage is successful, the persistently stored data block information in the Monitor-Result linked list is deleted.

[0209] In this example, the user re-uploads a data block of the file. The system confirms that the data block has been successfully re-uploaded and performs a comparison to ensure that there are no abnormalities in the data block. The system updates the information of the data block to the Monitor-Result linked list.

[0210] The system stores all successfully uploaded data blocks recorded in the Monitor-Result linked list in a persistent database. For example, the information of all data blocks of a file is stored in the database.

[0211] After the system confirms that the data block information has been successfully persisted and stored in the database, it deletes the successful data block information from the Monitor-Result linked list.

[0212] Example description:

[0213] Confirming the upload success of the data block: The user re-uploads a data block of the file. The system confirms that the data block has been uploaded successfully and there are no abnormalities, and updates its information to the Monitor-Result linked list.

[0214] Persistent storage of successful data block information: The system persistently stores all successfully uploaded data block information in the Monitor-Result linked list in the database to ensure that the information of these data blocks is preserved and managed for a long time.

[0215] Delete the successful data block information in the Monitor-Result linked list: After the system confirms that the data block information has been successfully persisted and stored in the database, it deletes the successful data block information from the Monitor-Result linked list to ensure that only the information that needs further processing is retained in the linked list.

[0216] Through the above steps, this embodiment can effectively manage the successfully uploaded data block information, ensure the long-term storage and management of data, and maintain the simplicity and accuracy of the Monitor-Result linked list.

[0217] Furthermore, a seventh embodiment of the distributed file upload verification method of the present invention is proposed. In the above step S10, the step of uploading the multiple data blocks to the distributed file system includes:

[0218] Step f1: uploading the multiple data blocks to the distributed file system using a multi-threaded uploading method.

[0219] In this embodiment, the file to be uploaded is first split into multiple data blocks according to a preset threshold value. These data blocks will be uploaded in parallel in subsequent steps.

[0220] To improve upload efficiency, the system uses multi-threaded uploads to upload multiple data blocks in parallel to the distributed file system. Each thread is responsible for uploading one or more data blocks. Multi-threading technology can maximize the use of network bandwidth and system resources.

[0221] When uploading a data block, each thread will record the upload status and related information in real time, such as the storage location of the data block, upload status, data block size, etc. This information will be stored in the Uploading linked list for subsequent integration and monitoring.

[0222] Example description:

[0223] Data block splitting: The user selected the file "large_file.zip". The system split it into five data blocks, each 20 MB.

[0224] Multi-threaded upload: The system starts 5 threads:

[0225] Thread 1 uploads data block 1.

[0226] Thread 2 uploads data block 2.

[0227] Thread 3 uploads data block 3.

[0228] Thread 4 uploads data block 4.

[0229] Thread 5 uploads data block 5.

[0230] Each thread uploads data blocks to the distributed file system in parallel.

[0231] Upload information record:

[0232] Thread 1 successfully uploads data block 1 and records the storage location, upload status, and size of data block 1.

[0233] Thread 2 successfully uploads data block 2 and records the storage location, upload status, and size of data block 2.

[0234] Similarly, the upload information of all data blocks is stored in the Uploading list.

[0235] Through the above steps, this embodiment can effectively utilize multi-threading technology to improve the efficiency of file uploading and ensure that the uploaded information of each data block is accurately recorded and managed, providing data support for subsequent integration and monitoring.

[0236] The present invention also provides a distributed file upload verification device.

[0237] Reference Figure 3 , Figure 3 This is a functional module diagram of the first embodiment of the distributed file upload verification device of the present invention. The distributed file upload verification device includes:

[0238] A file splitting and uploading module obtains a file to be uploaded, splits the file to be uploaded into multiple data blocks according to a preset threshold, uploads the multiple data blocks to a distributed file system, obtains upload information of each data block through the distributed file system, and stores the upload information of each data block in an Uploading linked list;

[0239] An upload information integration module integrates the upload information in the Uploading list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and uploads the integrated data to the Upload-Result list;

[0240] An upload status monitoring module monitors the integrated data of the Upload-Result linked list in real time, and when a successfully uploaded data block is detected, compares the successfully uploaded data block with the distributed file system;

[0241] The verification result prompt module prompts the user that the upload is abnormal if there is any abnormality in the successfully uploaded data block; and prompts the user that the upload is successful if there is no abnormality in all the multiple data blocks.

[0242] In addition, an embodiment of the present invention further provides a storage medium.

[0243] The storage medium stores a distributed file upload verification program, which implements the steps of the distributed file upload verification method described above when executed by the processor.

[0244] The specific implementation of the storage medium of the present invention is basically the same as the embodiments of the above-mentioned distributed file upload verification method, and will not be repeated here.

[0245] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of the present invention, or directly or indirectly used in other related technical fields, all fall within the protection of the present invention.

[0246] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0247] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0248] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0249] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A distributed file upload verification method, characterized in that: The distributed file upload verification method comprises the following steps: Obtain a file to be uploaded, split the file to be uploaded into multiple data blocks according to a preset threshold, upload the multiple data blocks to a distributed file system, obtain upload information of each data block through the distributed file system, and store the upload information of each data block in an Uploading linked list; Integrate the upload information in the Uploading linked list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and upload the integrated data to the Upload-Result linked list; monitoring the integrated data of the Upload-Result linked list in real time, and when detecting a successfully uploaded data block, comparing the successfully uploaded data block with the distributed file system; If the successfully uploaded data block has an abnormality, the user is prompted that the upload is abnormal; if all of the multiple data blocks have no abnormalities, the user is prompted that the upload is successful.

2. The distributed file upload verification method according to claim 1, wherein: After the step of if the successfully uploaded data block has an abnormality, the method further includes: The abnormal data blocks are deleted from the distributed file system.

3. The distributed file upload verification method according to claim 1, wherein: After the step of real-time monitoring the integrated data of the Upload-Result linked table, the method further includes: When it is detected that there is a data block that has failed to be uploaded, the data block that has failed to be uploaded is marked and the data block that has failed to be uploaded is deleted from the distributed file system.

4. The distributed file upload verification method according to claim 1, wherein: If the successfully uploaded data block has an abnormality, prompt the user to upload the abnormality; If there are no anomalies in the plurality of data blocks, after notifying the user that the upload is successful, the method further includes: Integrate all successfully uploaded data block information into the Monitor-Result linked list. When the user re-uploads the file to be uploaded, determine the data blocks to be uploaded that need to be re-uploaded according to the Monitor-Result linked list, and upload the data blocks to be uploaded to the distributed file system; If the data block to be uploaded is uploaded successfully, the user is prompted that the upload is successful.

5. The distributed file upload verification method according to claim 4, wherein: After the step of integrating the information of all successfully uploaded data blocks into the Monitor-Result linked list, the following steps are also included: Delete the upload information corresponding to all successfully uploaded data block information from the Upload-Result linked list.

6. The distributed file upload verification method according to claim 4, wherein: After the step of successfully uploading the data block to be uploaded, the method further includes: All successfully uploaded data block information in the Monitor-Result linked list is persistently stored in the database. If the persistent storage is successful, the persistently stored data block information in the Monitor-Result linked list is deleted.

7. The distributed file upload verification method according to claim 1, wherein: The step of uploading the plurality of data blocks to the distributed file system comprises: The multiple data blocks are uploaded to the distributed file system in a multi-threaded uploading manner.

8. A distributed file upload verification device, characterized in that: The distributed file upload verification device includes: A file splitting and uploading module obtains a file to be uploaded, splits the file to be uploaded into multiple data blocks according to a preset threshold, uploads the multiple data blocks to a distributed file system, obtains upload information of each data block through the distributed file system, and stores the upload information of each data block in an Uploading linked list; An upload information integration module integrates the upload information in the Uploading list to obtain integrated data, wherein the integrated data includes the upload information of each data block, and uploads the integrated data to the Upload-Result list; An upload status monitoring module monitors the integrated data of the Upload-Result linked list in real time, and when a successfully uploaded data block is detected, compares the successfully uploaded data block with the distributed file system; The verification result prompt module prompts the user that the upload is abnormal if there is any abnormality in the successfully uploaded data block; and prompts the user that the upload is successful if there is no abnormality in all the multiple data blocks.

9. A distributed file upload verification device, characterized in that: The distributed file upload verification device includes a memory, a processor, and a distributed file upload verification program stored in the memory and capable of running on the processor. When the distributed file upload verification program is executed by the processor, the steps of the distributed file upload verification method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that The storage medium stores a distributed file upload verification program, which, when executed by the processor, implements the steps of the distributed file upload verification method according to any one of claims 1 to 7.

Images