File fragment resuming method, device, equipment, medium and product

Abstract

The application provides a file fragment resuming method, device, equipment, medium and product, and relates to the field of big data. The method comprises the following steps: receiving a file fragment resuming request; in response to an authentication pass result of authentication information, calling a file list list from a receiving server, wherein the file list list comprises at least one file which is not uploaded completely and corresponds to the authentication information; in response to file selection information of the file list list, the file information carries a selected file which is not uploaded completely and first detail information thereof, and second detail information of the file which is not uploaded completely is called from the receiving server; when it is determined that the first detail information and the second detail information match successfully, file fragments which are not uploaded are resumed to the receiving server based on an un-uploaded fragment list of the file which is not uploaded completely. Through the above method, the consistency and integrity of the file fragment resuming are effectively guaranteed, and the ordered resuming function of the file fragment is realized.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a method, apparatus, device, medium and product for file segmented retransmission. Background Technology

[0002] The Hypertext Transfer Protocol (HTTP) is a request-response protocol commonly used for interaction between browsers and servers. When using a browser, users typically utilize HTTP technology to download files from or upload files to a server.

[0003] Currently, uploading large files using browser-based download technology often encounters problems such as slow upload speeds, file size limits imposed by gateways, and incomplete files due to lost data packets during HTTP transmission. Related technologies propose solutions that involve segmenting and transmitting large files to ensure file integrity and efficient transmission.

[0004] However, during file uploads, interruptions can occur due to browser crashes, network timeouts, or excessively long transmission times. Users may need to restart their browser or log in again to resume the upload. Currently, there is no mechanism for resuming file chunks, making it difficult for the system to determine which file and its corresponding chunk the user's request is for. This can lead to users resuming chunks of files they lack permission to upload after restarting their browser, resulting in a chaotic file chunk resuming process that compromises file transfer consistency and integrity. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for resuming file segmented transmission, in order to solve the problems of consistency and integrity in the process of resuming file segmented transmission.

[0006] According to a first aspect of this application, a method for resuming file segmented transmission is provided, comprising:

[0007] Receive a file fragment rescheduling request, wherein the file fragment rescheduling request carries authentication information;

[0008] In response to the authentication success result of the authentication information, a file list is retrieved from the receiving server, the file list including at least one file that has not been uploaded incompletely corresponding to the authentication information;

[0009] In response to the file selection information in the file list, the file information carries the selected incompletely uploaded file and its first details information, and the second details information of the incompletely uploaded file is retrieved from the receiving server;

[0010] When it is determined that the first details information and the second details information match successfully, the unuploaded file fragments are continued to be uploaded to the receiving server based on the list of unuploaded fragments of the unuploaded files.

[0011] In one implementation, the authentication information includes the requester's first account information.

[0012] The authentication success result in response to the authentication information includes:

[0013] Verify whether the first account information and the second account information used for uploading files are the same;

[0014] When the first account information and the second account information are the same, the authentication is successful in response to the authentication information.

[0015] In one implementation, determining that the first details information and the second details information match successfully includes:

[0016] The basic file information and corresponding file fragment information in the first details information are compared one by one with the basic file information and corresponding file fragment information in the second details information;

[0017] When the comparison results of the basic file information and the file fragment information are consistent, it is determined that the first detailed information and the second detailed information are successfully matched.

[0018] In one implementation, the basic file information includes at least one of the following: task identifier information, file size, and file information digest value; the file fragment information includes at least one of the following: fragment size, fragment transmission status, and fragment information digest value.

[0019] In one implementation, the file fragment retransmission request is a request from the terminal device to retransmit file fragments to the receiving server, and for the receiving server to dump the file fragments to the storage server in real time based on the SFTP session channel; the file fragment information includes fragment offset, fragment size, and fragment information digest value;

[0020] The step of comparing the basic file information and corresponding file fragment information in the first details information with the basic file information and corresponding file fragment information in the second details information one by one includes:

[0021] If the comparison results of the basic file information in the first details information and the basic file information in the second details information are consistent;

[0022] Then, based on the file basic information in the first details, the byte array of the corresponding segment is read, and the information digest processing value of the corresponding segment is obtained based on the byte array;

[0023] If the information digest processing value is consistent with the fragment information digest value corresponding to the basic file information in the second details information, then it is determined that the comparison result between the basic file information and the file fragment information is consistent.

[0024] In one embodiment, the method further includes:

[0025] When it is determined that the first details information and the second details information are successfully matched, the file fragments whose fragment transmission status is not completed in the first details information or the second details information are identified as file fragments to be resumed, and a list of unuploaded fragments is created based on the file fragments to be resumed.

[0026] In one implementation, after re-uploading the unuploaded file fragments to the receiving server based on the list of unuploaded file fragments, the method further includes:

[0027] Upon receiving the first verification pass indication, the verified file fragment is removed from the list of unuploaded fragments; wherein, the first verification pass indication is fed back to the terminal device by the receiving server when the corresponding file fragment has been re-uploaded and the message digest value of the file fragment has been verified.

[0028] And / or,

[0029] Upon receiving the second verification pass indication, the incomplete file that has passed verification is updated to an uploaded file; wherein, the second verification pass indication is provided by the receiving server to the terminal device when all file segments of the corresponding incomplete file have been re-uploaded and the total value of the information digest of the incomplete file has passed verification.

[0030] According to a second aspect of this application, a file segmentation and resuming device is provided, applied to a terminal device, comprising:

[0031] The request receiving module is configured to receive file segment resuming requests, wherein the file segment resuming requests carry authentication information.

[0032] The authentication retrieval module is configured to retrieve a file list from the receiving server in response to the authentication pass result of the authentication information. The file list includes at least one file that has not been uploaded in time and corresponds to the authentication information.

[0033] The module is configured to retrieve the second details of the incomplete file from the receiving server in response to the file selection information in the file list, wherein the file information carries the selected file that has not been uploaded and its first details information.

[0034] The segmented resume module is configured to resume uploading of unuploaded file segments based on the unuploaded segment list of unuploaded files when it is determined that the first details information and the second details information match successfully.

[0035] In one implementation, the authentication information includes the requester's first account information.

[0036] The authentication scheduling module includes: a verification unit, configured to verify whether the first account information and the second account information used for uploading files are the same; and a response unit, configured to respond to the authentication pass result of the authentication information when the first account information and the second account information are the same.

[0037] In one implementation, the segmented retransmission module includes:

[0038] The comparison unit is configured to compare the basic file information and corresponding file fragment information in the first details information with the basic file information and corresponding file fragment information in the second details information one by one;

[0039] A matching unit is configured to determine that the first details information and the second details information are successfully matched when the comparison results of the file basic information and the file fragment information are consistent.

[0040] According to a third aspect of this application, a terminal device is provided, comprising: a memory and a processor;

[0041] The memory stores computer-executed instructions;

[0042] The processor executes the computer execution instructions stored in the memory, causing the terminal device to execute the file segmentation and resuming method.

[0043] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, are used to implement the file segmentation and resuming method described above.

[0044] According to a fifth aspect of this application, a computer program product is provided, the computer program product including computer program code, which, when run on a computer, causes the computer to execute the file segmentation and resuming method.

[0045] Understandably, the file segmentation and resuming method, apparatus, device, medium, and product provided in this application embodiment receive a file segmentation and resuming request, which carries authentication information; in response to the authentication pass result of the authentication information, a file list is retrieved from the receiving server, which includes at least one file that has not been uploaded incompletely corresponding to the authentication information; in response to the file selection information in the file list, which carries the selected file that has not been uploaded incompletely and its first details information, the second details information of the file that has not been uploaded incompletely is retrieved from the receiving server; when it is determined that the first details information and the second details information match successfully, the file segments that have not been uploaded incompletely are resumed to the receiving server based on the list of file segments that have not been uploaded incompletely. In this process, authentication is performed on file segment resuming requests to limit and protect file resuming permissions. When authentication is successful, a file list is retrieved from the receiving server so that the user can select the corresponding incomplete file and its first detail information on the client. The first detail information is matched with the second detail information from the receiving server. When the match is successful, the file segment resuming process is started, which effectively ensures the consistency and integrity of file segment resuming and realizes the orderly resuming function of file segments. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating a file segmentation and resuming method provided in an embodiment of this application;

[0048] Figure 2 for Figure 1 A flowchart illustrating step S102;

[0049] Figure 3 for Figure 1 A flowchart illustrating step S103;

[0050] Figure 4 A flowchart illustrating another file segmentation and resuming method provided in an embodiment of this application;

[0051] Figure 5 A flowchart illustrating a file segmentation and resuming method provided for an exemplary embodiment of this application;

[0052] Figure 6 A schematic diagram of a file fragment retransmission device provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application.

[0054] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0055] In one possible application scenario, the system includes a terminal device, a receiving server, a scheduling server, and a storage server. These components are connected via wired or wireless networks. Optionally, there can be two or more receiving servers and storage servers, and their numbers can vary; for example, there could be three receiving servers and five storage servers. In this embodiment, a browser on the terminal device is used as the requesting end or client. The terminal device interacts with the receiving server via the network. Users can upload large files to the receiving server using Hypertext Transfer Protocol (HTTP) technology in their browser and initiate a file chunk retransmission request to the receiving server in case of file transfer interruption. Large files specifically refer to files that occupy a large amount of storage space, such as common video files or application software installation files. The scheduling server is used to receive the target storage server from multiple storage servers that will handle the transfer of the target file from the corresponding receiving server. After receiving the file chunk retransmission request, the terminal device verifies the request and, if the verification is successful, sends the file chunks to be retransmitted to the receiving server. The receiving server then transfers the retransmitted file chunks to the target storage server in real time.

[0056] The client devices may include, but are not limited to, computers, smartphones, tablets, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, portable computers, in-vehicle computers, wearable devices, desktop computers, set-top boxes, smart TVs, and so on.

[0057] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0058] It should be noted that the file fragmentation transmission method of this application can be used in the fintech field. It can also be used in any field other than fintech; the application field of the file fragmentation transmission method of this application is not limited.

[0059] In real-world applications, uploads may be interrupted due to browser crashes, network timeouts, or excessively long transmission times. Users need to restart their browser or log in again to resume the upload. However, since there is currently no mechanism for resuming file chunks, the system struggles to determine which file and its corresponding chunk the user's request is for. This can lead to users resuming chunks of files they don't have permission to upload after restarting their browser, resulting in a chaotic file chunk resuming process that compromises file transfer consistency and integrity.

[0060] In view of this, this embodiment provides a file segmented re-transmission method, apparatus, device, medium, and product. The method involves receiving a file segmented re-transmission request, the request carrying authentication information; responding to the authentication success result of the authentication information, retrieving a file list from a receiving server, the file list including at least one file corresponding to the authentication information that has not been fully uploaded; responding to file selection information in the file list, the file information carrying the selected file that has not been fully uploaded and its first details, retrieving second details of the file that has not been fully uploaded from the receiving server; and when it is determined that the first details and the second details match successfully, re-transmitting the unuploaded file segments to the receiving server based on the unuploaded segment list of the unuploaded files. In this process, authentication is performed on file chunk resume requests to limit and protect the resume permissions of general files. When authentication is successful, a file list is retrieved from the receiving server so that the user can select the corresponding incomplete file and its first detail information on the client. The first detail information is matched with the second detail information from the receiving server. When the match is successful, the file chunk resume process is started. This can effectively ensure the consistency and integrity of file chunk resume and realize the orderly resume function of file chunk.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0062] The application scenarios of this application have been briefly described above. The following will take the application to a terminal device as an example to explain in detail the file segmentation and resuming method provided by the embodiments of this application.

[0063] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a file segmentation and resuming method provided in an embodiment of this application, as shown below. Figure 1 As shown, it includes the following steps S101-S104.

[0064] Step S101: Receive a file segmentation resume request, wherein the file segmentation resume request carries authentication information.

[0065] In related technologies, resuming interrupted file transfers typically addresses the overall file transfer process. The main consideration is identifying and saving the breakpoint, then using that breakpoint to continue uploading during the resumption process. The overall file transfer usually doesn't present issues with consistency and integrity during resuming. However, when using chunked file uploads, without permission restrictions and upload detail verification during the resuming process, file chunks can easily be resent to other files that the uploader lacks the necessary permissions for. This can cause chaos in the resuming of file chunks and significantly compromise the consistency and integrity of the resuming process.

[0066] In this embodiment, when a user needs to resume file segment transmission, the user initiates a file segment transmission resume request to the terminal device. This file segment transmission resume request needs to carry the user's authentication information. When the terminal device successfully authenticates the user, it executes the relevant file transmission resume steps to ensure that the user has the permission to resume the file transmission and avoid file transmission disruptions.

[0067] Step S102: In response to the authentication pass result of the authentication information, retrieve the file list from the receiving server. The file list includes at least one file that has not been uploaded in time and corresponds to the authentication information.

[0068] Understandably, in response to, is used to indicate the conditions or states on which the operation being performed depends. When the conditions or states on which it depends are met, one or more operations being performed can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations being performed are executed.

[0069] In this embodiment, the terminal device can authenticate the authentication information, or it can receive authentication results (either successful or unsuccessful) from other terminals or servers. In response to a successful authentication result, the terminal device retrieves a file list from the receiving server. Specifically, the information in the receiving server's file list represents the receiving server's records and summaries of historically uploaded files. For example, the receiving server establishes task information for storing uploaded files and records the status of each uploaded file in a list format. The receiving server records the specific transmission details of each uploaded file in this database. When a file list retrieval request is received from the terminal device, at least one incompletely uploaded file within the authentication scope is sent to the terminal device based on the terminal device's authentication information.

[0070] In one implementation, a user account mechanism is used to authenticate the resume request and restrict and protect the file resume permission. Specifically, the authentication information includes the requester's first account information, such as... Figure 2 As shown, step S102, in response to the authentication pass result of the authentication information, may include the following steps S102a and S102b.

[0071] Step S102a: Verify whether the first account information and the second account information used for uploading files are the same;

[0072] Step S102b: When the first account information and the second account information are the same, respond to the authentication pass result of the authentication information.

[0073] In one implementation, a login page function for logging into user accounts can be added to the terminal device. When a user performs file (fragment) transfer and file (fragment) resume transfer, they first log in to the login page using their account information. In this way, the terminal device can record the account information (second account information) of the uploaded file when transferring files. For cases where multiple accounts upload files, the terminal device records all of them. When a user needs to resume file fragment transfer, they log in with their own account information (first account information) on the login page. The terminal device first verifies the first account information with the second account information of the historical uploaded files recorded by the terminal device. If there are multiple second account information for historical uploaded files, each of the multiple second account information can be verified with the first account information. If the first second account information is the same as the first account information, the authentication information passes.

[0074] For example, at least one file that has not been uploaded within the scope of the above authentication, i.e., when the first account information and the second account information are the same, is an incomplete file generated by uploading a file based on the second account information.

[0075] In this process, user accounts are used to authenticate and verify file chunk resuming requests. This quickly verifies whether the user has previously uploaded the file, ensuring the security of file chunk resuming and reducing confusion during the process. Furthermore, the authentication process is efficient and convenient. In some embodiments, in addition to the above authentication methods, other authentication information, such as encryption keys, can also be used.

[0076] It should be noted that the first account information and the second account information in this embodiment are only used to distinguish similar objects and have no other special meaning. They can be the same content or different content.

[0077] Step S103: In response to the file selection information in the file list, the file information carries the selected file that has not been uploaded and its first details information, and retrieves the second details information of the file that has not been uploaded from the receiving server.

[0078] In this embodiment, after the terminal device receives the file list from the receiving server, the user can select one or more unuploaded files from the file list on the terminal device's resume upload page to continue the upload. For each of these unuploaded files, the terminal device retrieves its corresponding first details. Specifically, the terminal device also retains file upload details during the upload process.

[0079] Optionally, the user selects the file to be uploaded from the file list on the local computer where the browser is located, based on the returned file list. The browser-side web application responds to the onchange event of the selected file, obtains access permissions to the file's binary Blob object, obtains the first detailed information of the corresponding file locally, and sends a query request to the server for the second detailed information of the file (this request may include the corresponding file's TASK_ID information). The server responds to the browser's request, queries and returns the detailed information of the corresponding file's upload status (i.e., the second detailed information) based on the received TASK_ID.

[0080] The first details and / or the second details may include, but are not limited to, the following: basic file information and file fragment information. The basic file information may include one or a combination of the following: task identifier (TASK_ID), file size (FILE_SIZE), and file information digest value MD5. The file fragment information may include one or a combination of the following: fragment size (SLICE_SIZE), fragment transfer status (STATE), and fragment information digest value (MD5_SLICE).

[0081] It should be noted that the above information is merely an example of this embodiment and is not intended to be limiting. In some embodiments, the basic file information may also include the MD5 value of the first 1024 bytes of the file (MD5_1024), the size of the transmitted file (RECEIVED_TOTAL_SIZE), and the file fragment information may also include: fragment offset (SLICE_OFFSET), etc.

[0082] Furthermore, this embodiment can effectively solve the problem of difficulty in quickly finding the file to continue uploading when a user logs back in after multiple file transfers are interrupted, by retrieving the file list from the receiving server and selecting the file to be resumed. This further improves the efficiency of file chunked resume transfer.

[0083] In one implementation, the first and second details information contain multiple types of information. By comparing each type of information one by one, the accuracy of the details information verification is improved. Specifically, step S103 determines that the first and second details information match successfully, such as... Figure 3 As shown, the following steps may be included:

[0084] Step S103a: Compare the basic file information and corresponding file fragment information in the first details information with the basic file information and corresponding file fragment information in the second details information one by one;

[0085] Step S103b: When the comparison results of the basic file information and the file fragment information are consistent, it is determined that the first details information and the second details information are successfully matched.

[0086] In one implementation, to further improve the efficiency of verifying detailed information, and considering the verification process when the file to be resumed is a file that needs to be real-time dumped using the Secure File Transfer Protocol (SFTP), in this embodiment, the file fragment resume request is a request from the terminal device to resume the file fragment to the receiving server, and for the receiving server to real-time dump the file fragment to the storage server based on the SFTP session channel; the file fragment information includes fragment offset, fragment size, and fragment information digest value.

[0087] Understandably, SFTP provides a secure network encryption method for file transfer. Related technologies utilize SFTP's append access mode. By creating a storage file of a specified size on the storage server (used for writing file fragments), and setting a negative offset in the form of absolute offset = offset (actual offset) - size, random access to the storage file is indirectly achieved, thus enabling file fragments to be written to the specified offsets of the storage file on the storage server. In this embodiment, under this file fragment transmission method, the file fragment information may include fragment offset information. The fragment offset information and fragment size can be used to calculate a local MD5 hash (i.e., the fragment MD5 value on the terminal device), and this local MD5 hash is compared and verified with the MD5 hash of the receiving client to verify the fragment details information in the SFTP dump format.

[0088] The file segment information can be either file segments that have been uploaded but have not yet been uploaded.

[0089] Step S103a involves comparing the basic file information and corresponding file fragment information in the first details information with the basic file information and corresponding file fragment information in the second details information one by one. This may include the following steps:

[0090] If the comparison results of the basic file information in the first details information and the basic file information in the second details information are consistent;

[0091] Then, based on the file basic information in the first details, the byte array of the corresponding segment is read, and the information digest processing value of the corresponding segment is obtained based on the byte array;

[0092] If the information digest processing value is consistent with the fragment information digest value corresponding to the basic file information in the second details information, then it is determined that the comparison result between the basic file information and the file fragment information is consistent.

[0093] For example, the basic file information is first verified: for file size verification, the file size selected by the user should be consistent with the file size returned by the server; for file MD5_1024 verification (if there is no complete MD5 record for the file), the local MD5 digest value of the first 1024 bytes of the file selected by the user should be consistent with the MD5_1024 returned by the server; for complete file MD5 verification (if there is no record in the details, this can be skipped), if the server returns the complete file MD5, the local complete MD5 digest value of the file selected by the user should be consistent with the MD5_TOTAL returned by the server. After the basic file information verification passes, the file fragment information is further verified. If the basic file information verification fails, a verification failure message is sent, instructing the user to reselect the file to be uploaded. The byte array of the fragment is read based on the fragment offset (SLICE_OFFSET) and fragment size (SLICE_SIZE), and the local MD5 digest of the fragment is calculated. This digest is then compared with the digest value returned by the receiving server. If any of the above information comparisons are inconsistent, the user is prompted that the candidate file does not match the original uploaded file and needs to reselect or exit. Once all the above information has been verified and found to be consistent, the first and second details information are confirmed to be a successful match.

[0094] Step S104: When it is determined that the first details information and the second details information are successfully matched, the unuploaded file fragments are continued to be uploaded to the receiving server based on the unuploaded fragment list of the unuploaded files.

[0095] In this embodiment, the above-mentioned verification method is used to match and verify the details of each unuploaded file. For multiple unuploaded files that pass the verification, the incomplete file segments of multiple unuploaded files can be resumed at the same time, which effectively improves the efficiency of file resume transmission.

[0096] Optionally, in a file transfer scenario where file fragments are dumped in real time based on SFTP technology, the terminal device sequentially transmits the unuploaded file fragments to the receiving server according to the list of unuploaded fragments. After receiving the file fragment, the receiving server verifies the digest value of the file fragment, and when the verification passes, dumps the file fragment to the specified offset of the storage file on the storage server in real time based on SFTP technology.

[0097] Furthermore, this embodiment further enables the orderly resumption of unuploaded file fragments by identifying file fragments with incomplete transmission status as file fragments to be resumed and creating a list of unuploaded fragments. Specifically, the method may also include the following steps:

[0098] When it is determined that the first details information and the second details information are successfully matched, the file fragments whose fragment transmission status is not completed in the first details information or the second details information are identified as file fragments to be resumed, and a list of unuploaded fragments is created based on the file fragments to be resumed.

[0099] In this embodiment, when the receiving server (or terminal device) receives (sends) file fragments of historically uploaded files, it can record the fragment transmission status of the file fragments in the first details information (or the second details information). When the terminal device has a need to resume the transmission of file fragments, it determines the file fragments marked as completed as file fragments to be resumed, creates a corresponding list of unuploaded fragments, and resumes the transmission of file fragments sequentially based on the list of unuploaded fragments.

[0100] Alternatively, please refer to Figure 4 , Figure 4 This is a flowchart illustrating another file chunk retransmission method provided in this application embodiment. By deleting relevant information in the list or updating the file upload status when file chunks and / or file transmission is successful, multiple uploads of the same file chunk and / or the same file can be effectively avoided, facilitating file chunk transmission management. Specifically, after step S104 retransmits the unuploaded file chunks to the receiving server based on the list of unuploaded chunks of the unuploaded file, the following steps may also be included:

[0101] Step S401: Upon receiving the first verification pass indication, the verified file fragment is deleted from the list of unuploaded fragments; wherein, the first verification pass indication is fed back by the receiving server to the terminal device when the corresponding file fragment has been re-uploaded and the information digest value of the file fragment has been verified.

[0102] Step S402: Upon receiving the second verification pass indication, update the verified incomplete upload files to uploaded files; wherein, the second verification pass indication is provided by the receiving server to the terminal device when all file segments of the corresponding incomplete upload file have been re-uploaded and the total value of the information digest of the incomplete upload file has been verified.

[0103] It should be noted that steps S401 and S402 can be chosen individually or in combination. In the example of combination, steps S401 and S402 have a fixed order and can be executed in parallel or sequentially. This embodiment does not make specific limitations on this.

[0104] To facilitate understanding of the technical solution of this application, combined with Figure 5 As shown, this exemplary embodiment will be described in conjunction with the client, receiving server, scheduling server, and storage server, and includes the following process:

[0105] Step S501: The terminal device initiates a file fragment retransmission request to the receiving server, requesting the receiving server to provide a list of incomplete files. This request carries the user's account information or other authentication information.

[0106] It should be noted that during the initial transmission phase of file chunk retransmission (i.e., before file chunk transmission is interrupted), the receiving server records the task (details) information of the file chunk transmission to facilitate resuming transmission if file chunk transmission is interrupted. This task details may include, but are not limited to, the following: MD5 digest information for the first 1024 bytes of the file (MD5_1024); file size: FILE_SIZE; file task identifier: TASK_ID; complete MD5 digest information for the file: MD5_TOTAL (this information can be obtained by the receiving server after the terminal device has calculated it and sent it via an asynchronous interface); a list of chunks received by the receiving server; detailed information for each received chunk (chunk list information); chunk offset (SLICE_OFFSET); chunk size (SLICE_SIZE); chunk MD5 value (MD5_SLICE); chunk storage completion status on the storage server (STATE = [COMPLETE|INIT|ABORT|TIMEOUT]); the size of the file received by the receiving server (RECEIVED_TOTAL_SIZE); and the user account of the user who uploaded the file (temporary file owner).

[0107] In some possible examples, the above information is recorded in the receiving server's database during the initial upload phase, the full MD5 hash of the file being sent, and the fragmented upload phase, respectively.

[0108] Step S502: Receive the list of incomplete files from the server to the terminal device.

[0109] Step S503: The terminal device opens the resume page based on the file list returned by the receiving server.

[0110] Step S504: The user selects the local file to resume download on the resume page.

[0111] Step S505: The JS script on the terminal device obtains the file selected by the user for resuming download through the onchange event.

[0112] Step S506: The terminal device sends a file details information request to the receiving server. The request parameters may include the file's task identifier (TaskID) and user account information.

[0113] Step S507: Receive the server's response to the file details information request and return the task details information (i.e., the second details information) of the specified file.

[0114] Step S508: The terminal device reads the binary information of the selected file, obtains access permissions for the binary Blob object of the file, and reads the local (task) details information (i.e., the first details information) corresponding to the file.

[0115] Step S509: The terminal device verifies the first details information and the second details information. If the verification is successful, proceed to step S511 to resume file transfer; otherwise, return to step S504 to select a file to resume file transfer.

[0116] Optionally, the terminal device first determines whether to reselect the file to continue the transfer. If yes, it returns to step S504; otherwise, the process ends.

[0117] Step S511: The terminal device creates a list of unuploaded fragments.

[0118] Step S512: The terminal device sends a file resume request to the receiving server.

[0119] Step S513: Receive the task information (which may include TaskID, file MD5_1024, and a list of unuploaded fragments) recorded by the server and store it in the local database.

[0120] Step S514: The receiving server sends a dump request to the scheduling server, which carries the file MD5_1024.

[0121] Step S515: The scheduling server matches the corresponding storage server based on the MD5_1024 file.

[0122] In one implementation, the scheduling server can query whether a corresponding storage server has been matched based on the TaskID of the file. If so, the matched storage server information is sent to the receiving server; otherwise, a storage server is allocated according to the preset configuration information and the MD5 hash of the file (MD5_1024). The preset configuration information is a directory structure table about MD5 pre-created by the scheduling server. This directory structure table contains 256*256 directories formed by possible combinations of the first four digits of the MD5 hash, and associates the directories in the target structure table with multiple storage servers in the system.

[0123] Step S516: The scheduling server sends the matched storage server information back to the receiving server, and the receiving server records the storage server assigned to it by the scheduling server.

[0124] In one implementation, instead of the above steps S514-S516, the receiving server can retrieve the historical task information of the file from the local database based on the file's TaskID, and determine whether the scheduling server has allocated a storage server for the file based on the file's MD5_1024. If so, there is no need to send a request to the scheduling server again, and the file fragments can be directly dumped based on the allocated storage server when the file fragments are received.

[0125] Step S517: The terminal device sends file fragment data to the receiving server based on the list of unuploaded fragments. Execution conditions are set: determine if any unsent fragments exist; if so, proceed to step S518; otherwise, proceed to step S530.

[0126] Step S518: The terminal device reads the file fragments and calculates the file fragment MD5, file fragment SIZE, etc.

[0127] Step S519: The terminal device transmits file fragments to the receiving server in the form of an HTTP PUT request.

[0128] Step S520: The receiving server performs MD5 verification on the file fragments to verify whether the MD5 of the received file fragments is consistent with the MD5 of the file fragment source.

[0129] Step S521: After the MD5 value verification of the file fragment passes, the receiving server locks the current (storage) file creation (operation) permissions through a global lock control (e.g., redission-rlock).

[0130] Step S522: The storage server confirms the file creation status.

[0131] Step S523: The receiving server determines whether the current file has been created based on the file creation status of the storage server. If the file has not been created, the receiving server is instructed to create the file. If the file has been created, step S524 is executed.

[0132] Specifically, the file can call the SFTP write method with a specified offset to create a file of a specified size, and the storage file size is determined based on the size of the passed-in file.

[0133] Step S524: The storage server creates a file according to the instructions of the receiving server.

[0134] Step S525: Unlock the creation permission of the current file and lock the write (write operation) permission of the current file fragment.

[0135] Locking file creation permissions means locking the global file creation permission for the MD5 digest value of the source file; unlocking file creation operation permissions means unlocking the global file creation permission for the MD5 digest value of the source file.

[0136] Step S526: With write permissions for file fragments, the storage server writes file fragments into the storage file.

[0137] Step S527: Unlock write permissions for the current file segment to facilitate writing to subsequent file segments.

[0138] Understandably, steps S505-S525 above are the fragmentation transmission and dumping stages.

[0139] Step S528: During the process of segmented retransmission, the terminal device calculates the complete MD5 value of the file in parallel.

[0140] In one implementation, the terminal device first retrieves the complete MD5 value from the receiving server. If the retrieval is successful, it means that the complete MD5 value of the file has been calculated during the file upload stage, and it is not necessary to calculate the complete MD5 value of the file again during the file segmentation and retransmission stage. If the retrieval fails, that is, the receiving server does not record the complete MD5 value of the file, it means that the complete MD5 value of the file has not been calculated during the file upload stage.

[0141] Step S529: The terminal sends a file MD5 setting request (upload / md5) to the receiving server to synchronize the complete MD5 value of the target file. The receiving server receives the complete MD5 value of the target file and sets the complete MD5 value of this upload task in the server database. The request method can be a POST request.

[0142] In this embodiment, a parallel process for the client to calculate the complete MD5 of a file is created. This process is synchronized with the file chunk upload after the client responds to the server to create the file (upload / begin). The complete MD5 value of the file is calculated and sent to the receiving server.

[0143] Understandably, steps S528 and S529 above constitute the complete MD5 synchronization phase of the file.

[0144] Step S530: Wait at the parallel business aggregation node for all preceding parallel processes (file fragmentation and transmission completed, MD5 calculation completed) to be completed.

[0145] The client's parallel process aggregation node waits for the file slice upload and file MD5 setting to be completed by setting a cyclic delay waiting mechanism on the client side, and then enters the end stage of the process.

[0146] Step S531: The terminal sends a request to the receiving server: Step 3: upload / finish request (file fragment transfer completion request).

[0147] Understandably, when both independent parallel processes have their end flags set, the client triggers the termination of the process.

[0148] Step S532: The receiving server sends a message to the storage server to calculate the total MD5 hash of the file fragments. At the same time, it retrieves the complete MD5 hash of the target file sent by the client from the local database, verifies whether they are consistent, and returns the result of successful or failed transmission of the target file to the client.

[0149] Step S533: Process terminated.

[0150] In practical applications, resuming file transfers in scenarios such as browser crashes has always been a challenging issue in the industry. While publicly available business scenarios employ idempotent resuming mechanisms based on Uniform Resource Locator (URL) address consistency, these mechanisms lack protection for the consistency and integrity of the uploaded files. This embodiment utilizes a user account mechanism to restrict and protect resuming permissions for general files. It ensures the consistency between the resuming file and the original file through task details and a series of MD5 digest checks. Combined with an HTTP chunked upload mechanism, it implements the final data transmission steps for resuming the file transfer, effectively solving the problems of file transmission integrity and consistency in the HTTP resuming process. Furthermore, it possesses broad applicability and versatility.

[0151] Please refer to Figure 6 , Figure 6 This is a schematic diagram of a file segmented resuming transfer device provided in an embodiment of this application. Applied to a terminal device, the device includes a request receiving module 61, an authentication retrieval module 62, a selection retrieval module 63, and a segmented resuming transfer module 64.

[0152] The request receiving module 61 is configured to receive file segment resuming requests, wherein the file segment resuming requests carry authentication information.

[0153] The authentication retrieval module 62 is configured to retrieve a file list from the receiving server in response to the authentication pass result of the authentication information. The file list includes at least one file that has not been uploaded in time and is corresponding to the authentication information.

[0154] Select retrieval module 63 is configured to retrieve second details of the incompletely uploaded file from the receiving server in response to file selection information in the file list, wherein the file information carries the selected file that has not been uploaded and its first details information.

[0155] The segmented resume transmission module 64 is configured to resume transmission of unuploaded file segments based on the unuploaded segment list of unuploaded files when it is determined that the first details information and the second details information match successfully.

[0156] In one implementation, the authentication information includes the requester's first account information.

[0157] The authentication scheduling module 62 includes: a verification unit, configured to verify whether the first account information and the second account information used for uploading files are the same; and a response unit, configured to respond to the authentication pass result of the authentication information when the first account information and the second account information are the same.

[0158] In one embodiment, the segmented retransmission module 64 includes:

[0159] The comparison unit is configured to compare the basic file information and corresponding file fragment information in the first details information with the basic file information and corresponding file fragment information in the second details information one by one;

[0160] A matching unit is configured to determine that the first details information and the second details information are successfully matched when the comparison results of the file basic information and the file fragment information are consistent.

[0161] In one implementation, the basic file information includes at least one of the following: task identifier information, file size, and file information digest value; the file fragment information includes at least one of the following: fragment size, fragment transmission status, and fragment information digest value.

[0162] In one implementation, the file fragment retransmission request is a request from the terminal device to retransmit file fragments to the receiving server, and for the receiving server to dump the file fragments to the storage server in real time based on the SFTP session channel; the file fragment information includes fragment offset, fragment size, and fragment information digest value;

[0163] The comparison unit is specifically configured to: if the comparison results of the file basic information in the first details information and the file basic information in the second details information are consistent; then read the byte array of the corresponding segment based on the segment offset and segment size corresponding to the file basic information in the first details information, and obtain the information digest processing value of the corresponding segment based on the byte array; if the information digest processing value is consistent with the segment information digest value corresponding to the file basic information in the second details information, then determine that the comparison results of the file basic information and the file segment information are consistent.

[0164] In one embodiment, the apparatus further includes:

[0165] The list creation module is configured to, when it is determined that the first details information and the second details information match successfully, identify the file fragments whose fragment transmission status is incomplete as pending file fragments in the first details information or the second details information as pending file fragments, and create a list of unuploaded fragments based on the pending file fragments.

[0166] In one embodiment, the apparatus further includes:

[0167] The deletion module is configured to delete the verified file fragment from the list of unuploaded fragments upon receiving a first verification pass indication; wherein, the first verification pass indication is fed back to the terminal device by the receiving server when the corresponding file fragment has been resumed and the message digest value of the file fragment has been verified.

[0168] And / or,

[0169] The update module is configured to update the verified incomplete upload files to uploaded files when a second verification pass indication is received; wherein, the second verification pass indication is provided by the receiving server to the terminal device when all file segments of the corresponding incomplete upload file have been re-uploaded and the total value of the information digest of the incomplete upload file has been verified.

[0170] It should be noted that the device provided in this application can implement all the method steps implemented by the terminal device in the above method embodiment and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0171] This application also provides a terminal device, such as... Figure 7 As shown, the terminal device may include: transceiver 71, processor 72, and memory 73.

[0172] The processor 72 executes computer execution instructions stored in the memory, causing the processor 72 to execute the file segmentation and continuation method provided in the above embodiments. The processor 72 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0173] The memory 73 is connected to the processor 72 via the system bus and completes communication between them. The memory 73 is used to store computer program instructions.

[0174] Transceiver 71 can be used to obtain the task to be run and its configuration information.

[0175] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0176] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the file segmentation and continuation method.

[0177] It should be noted that the computer-readable storage medium provided in this application can implement all the method steps implemented by the terminal device in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0178] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute the file segmentation and resuming method.

[0179] It should be noted that the above-mentioned program product provided in this application can correspondingly implement all the method steps implemented by the terminal device in the above method embodiment, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0180] This application also provides a chip, including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory to execute the file segmentation and continuation method.

[0181] It should be noted that the chip provided in this application can implement all the method steps implemented by the terminal device in the above method embodiment and achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail here.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0183] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0184] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0185] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0186] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0187] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0188] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0189] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0190] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0191] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0192] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0193] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A file fragment resuming method, characterized in that, Applied to a terminal device, comprising: receiving a file segment resuming request, the file segment resuming request carrying authentication information; in response to the authentication passing result of the authentication information, calling a file list manifest from a receiving server, the file list manifest including at least one file that has not been uploaded completely corresponding to the authentication information; in response to file selection information of the file list manifest, the file selection information carrying a selected file that has not been uploaded completely and first detail information thereof, calling second detail information of the file that has not been uploaded completely from the receiving server; the first detail information is file upload detail information reserved by the terminal device when uploading the file; when it is determined that the first detail information and the second detail information match successfully, resuming the file segment that has not been uploaded to the receiving server based on an un-uploaded segment list of the file that has not been uploaded completely; the determination that the first detail information and the second detail information match successfully comprises: respectively comparing file basic information and corresponding file segment information in the first detail information with file basic information and corresponding file segment information in the second detail information one by one; when the comparison results of the file basic information and the file segment information are consistent, determining that the first detail information and the second detail information match successfully; the file basic information includes the following information: task identification information, file size and file information digest value; the file segment information includes the following information: segment size, segment transmission state and segment information digest value; after resuming the file segment that has not been uploaded to the receiving server based on the un-uploaded segment list of the file that has not been uploaded completely, further comprising: when a first check passing indication is received, deleting the file segment that has passed the check from the un-uploaded segment list; wherein the first check passing indication is fed back by the receiving server to the terminal device when the corresponding file segment resuming is completed and the information digest value of the file segment passes the check; when a second check passing indication is received, updating the file that has not been uploaded completely that has passed the check to an uploaded file; wherein the second check passing indication is fed back by the receiving server to the terminal device when all file segments of the corresponding file that has not been uploaded completely are resumed and the information digest total value of the file that has not been uploaded completely passes the check.

2. The method of claim 1, wherein, the authentication information includes first account information of a requestor, then in response to the authentication passing result of the authentication information, comprising: checking whether the first account information and second account information for uploading the file are the same; when the first account information and the second account information are the same, in response to the authentication passing result of the authentication information.

3. The method of claim 1, wherein, the file segment resuming request is a request of resuming a file segment to a receiving server by the terminal device, and making the receiving server real-time dump the file segment to a storage server based on an SFTP session channel; the file segment information includes segment offset, segment size and segment information digest value; The one-to-one comparison of the file basic information and the corresponding file fragment information in the first detail information with the file basic information and the corresponding file fragment information in the second detail information comprises: If the comparison result of the file basic information in the first detail information and the file basic information in the second detail information is consistent; Then, the byte array of the corresponding fragment is read based on the fragment offset and the fragment size of the file fragment information in the first detail information, and the information digest processing value of the corresponding fragment is obtained based on the byte array; If the information digest processing value is consistent with the fragment information digest value corresponding to the file fragment information in the second detail information, it is determined that the comparison result of the file basic information and the corresponding file fragment information in the first detail information with the file basic information and the corresponding file fragment information in the second detail information is consistent.

4. The method according to any one of claims 1 to 3, characterized in that, Further comprising: When it is determined that the first detail information and the second detail information match successfully, the file fragment whose transmission state is identified in the first detail information or the second detail information as not being transmitted completely is determined as a file fragment to be continued to transmit, and an un-uploaded fragment list is created based on the file fragment to be continued to transmit.

5. A file fragment resuming apparatus characterized by comprising: Applied to a terminal device, comprising: A request receiving module is configured to receive a file fragment continuous transmission request, and the file fragment continuous transmission request carries authentication information; An authentication calling module is configured to call a file list from a receiving server in response to the authentication passing result of the authentication information, and the file list includes at least one file which is not uploaded completely and corresponds to the authentication information; A selection calling module is configured to call second detail information of the file which is not uploaded completely from the receiving server in response to file selection information of the file list, and the file selection information carries the selected file which is not uploaded completely and first detail information of the file; A fragment continuous transmission module is configured to continue to transmit the file fragment which is not uploaded based on an un-uploaded fragment list of the file which is not uploaded completely when it is determined that the first detail information and the second detail information match successfully. The fragment continuous transmission module comprises: A comparison unit is configured to compare the file basic information and the corresponding file fragment information in the first detail information with the file basic information and the corresponding file fragment information in the second detail information one by one; A matching unit is configured to determine that the first detail information and the second detail information match successfully when the comparison result of the file basic information and the file fragment information is consistent. The file basic information includes task identification information, file size and file information digest value, and the file fragment information includes fragment size, fragment transmission state and fragment information digest value; The device further comprises: a deleting module configured to delete the file fragment that passes the check from the un-uploaded fragment list upon receiving a first check pass indication; wherein the first check pass indication is fed back by the receiving server to the terminal device when all the file fragments corresponding to the file fragment are uploaded and the information digest value of the file fragment passes the check; an updating module configured to update the un-uploaded file that passes the check to an uploaded file upon receiving a second check pass indication; wherein the second check pass indication is fed back by the receiving server to the terminal device when all the file fragments corresponding to the un-uploaded file are uploaded and the information digest total value of the un-uploaded file passes the check.

6. The apparatus of claim 5, wherein, The authentication information includes first account information of the requestor, The authentication scheduling module includes: a checking unit configured to check whether the first account information and second account information for uploading the file are the same; and a responding unit configured to respond to an authentication pass result of the authentication information when the first account information and the second account information are the same.

7. A terminal device, characterized by, comprising: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the terminal device executes the file fragment resuming method in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the file fragment resuming method in any one of claims 1 to 4.

9. A computer program product, characterised in that, The computer program product includes computer program code, and when the computer program code runs on the computer, the computer program code makes the computer execute the file fragment resuming method in any one of claims 1 to 4.

Images