File fragment transmission method, device, equipment, medium, product and system

Abstract

The application provides a file fragment transmission method, device, equipment, medium, product and system, and relates to the field of big data. The method comprises the following steps: determining a target byte number according to a target file size, and obtaining an information digest value of the target byte number; initiating a transmission request of the target file to a receiving server, the transmission request carrying the information digest value, so that the receiving server obtains the information digest value, initiates a dump request to a scheduling server, and the scheduling server matches a target storage server from a plurality of storage servers according to the information digest value; and transmitting file fragments of the target file to the receiving server, so that the file fragments are sequentially dumped to the target storage server, and the target storage server combines all the file fragments to obtain the target file. Through the above method, the application solves the delay and congestion problems generated in the initial stage of the file uploading process, and effectively improves the file uploading efficiency.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a method, apparatus, device, medium, product and system for file fragmentation and transmission. Background Technology

[0002] The Hypertext Transfer Protocol (HTTP) is a request-response protocol commonly used for interaction between browsers and servers. Users typically utilize HTTP technology to download files from or upload files to servers when using a browser.

[0003] During file upload, after one or more servers receive a large file locally, they usually need to dump it to a remote dedicated storage server. Since the storage capacity of each storage server is limited, it may be difficult to meet the dumping needs of the server when there are many files. In related technologies, multiple storage servers are scheduled and the corresponding target storage server is assigned to the file to realize the file dumping, thereby greatly improving the overall throughput and storage capacity of the system.

[0004] However, the dumping process can only be carried out after a corresponding target storage server is allocated to the file. Since the allocation of the target storage server in related technologies depends on the client calculating the complete message-digest algorithm 5 (MD5) value of the file, and for large files, especially those 4GB or larger, the calculation of the digest value requires a long time, the allocation time of the target storage server is relatively long. This results in a large delay and blockage in the initial stage of the file dumping process, which in turn affects the efficiency of file upload to a certain extent. Summary of the Invention

[0005] This application provides a file fragmentation transmission method, apparatus, device, medium, product, and system to at least solve the above-mentioned technical problems.

[0006] According to a first aspect of this application, a file fragmentation transmission method is provided, applied to a client, the method comprising:

[0007] The target number of bytes is determined based on the target file size, and the information digest value of the target number of bytes is obtained;

[0008] A transmission request for the target file is initiated to the receiving server. The transmission request carries the information digest value, so that the receiving server can obtain the information digest value and initiate a dump request to the scheduling server. The scheduling server then matches a target storage server from multiple storage servers based on the information digest value to receive the target file dumped by the receiving server.

[0009] The file fragments of the target file are transmitted sequentially to the receiving server, so that when the receiving server receives the file fragments, it sequentially dumps the file fragments to the target storage server, and the target storage server combines all the file fragments to obtain the target file after all the file fragments of the target file have been transmitted.

[0010] In one embodiment, the method further includes:

[0011] Obtain the total first message digest value of the target file and transmit the total first message digest value to the receiving server;

[0012] The step of having the target storage server combine all file fragments to obtain the target file after all file fragments of the target file have been transmitted includes:

[0013] When the receiving server receives a verification instruction, it verifies whether the total value of the first message digest and the total value of the second message digest of all file fragments are consistent. If they are consistent, it triggers the combination of all file fragments in the target storage server to obtain the target file. In one embodiment, determining the target number of bytes based on the target file size includes:

[0014] When the target file size is greater than or equal to a preset number of bytes, the first N bytes of the target file are determined as the target number of bytes; where N is a positive integer.

[0015] When the target file size is less than the preset number of bytes, the total number of bytes in the target file is determined as the target number of bytes.

[0016] In one implementation, the transfer request also carries a unique identifier for the target file and the size of the target file.

[0017] In one implementation, after sequentially transmitting file fragments to the receiving server, the method further includes:

[0018] If an update instruction is received from the receiving server, the updated file fragment corresponding to the file fragment is resent to the receiving server; wherein, the update instruction is issued by the receiving server when it verifies that the message digest value of the file fragment is inconsistent with the preset message digest value, and is used to instruct the resentment of the updated file fragment of the corresponding file fragment.

[0019] In one implementation, the transmission request is in the form of an HTTP POST request, and the file fragmentation request is in the form of an HTTP PUT request.

[0020] According to a second aspect of this application, another file fragmentation transmission method is provided, applied to a receiving server, the method comprising:

[0021] After receiving a transmission request for a target file initiated by a client, a dump request is sent to the scheduling server. The transmission request carries a message digest value of the target file. The message digest value is obtained based on the target number of bytes after determining the target file size. This allows the scheduling server to match a target storage server from multiple storage servers based on the message digest value to receive the target file dumped by the receiving server.

[0022] The system receives file fragments of the target file transmitted by the client and dumps the file fragments to the target storage server, so that the target storage server can combine all file fragments to obtain the target file after all file fragments of the target file have been transmitted.

[0023] In one embodiment, the method further includes:

[0024] Receive the total first message digest value of the target file transmitted by the client;

[0025] Upon receiving a verification instruction, the system verifies whether the total value of the first message digest and the total value of the second message digest of all file fragments are consistent; wherein, the verification instruction is issued by the client in response to the completion of transmission of all file fragments of the target file;

[0026] The step of having the target storage server combine all file fragments to obtain the target file after all file fragments of the target file have been transmitted includes:

[0027] When the total value of the first message digest and the total value of the second message digest are the same, all file fragments in the target storage server are combined to form the target file. In one embodiment, the transmission request also carries a unique identifier of the target file and the size of the target file;

[0028] Then, after receiving the target file transfer request initiated by the client and before initiating a dump request to the scheduling server, the method further includes:

[0029] Task information is generated based on the unique identifier of the target file, the size of the target file, and the message digest value, and the task information is stored in the target database;

[0030] The step of initiating a dump request to the scheduling server includes: retrieving the task information from the target database, obtaining the information digest value corresponding to the target file based on the task information, and initiating a dump request to the scheduling server.

[0031] In one embodiment, the method further includes:

[0032] The message digest value of the file fragment is checked to see if it is consistent with the preset message digest value. If they are inconsistent, an update instruction is sent to the client. The update instruction is used to instruct the client to resend the updated file fragment of the corresponding file fragment.

[0033] According to a third aspect of this application, a file fragmentation transmission apparatus is provided, applied to a client, the apparatus comprising:

[0034] The acquisition module is configured to determine the target number of bytes based on the target file size and acquire the information digest value of the target number of bytes;

[0035] The first request module is configured to initiate a transmission request for the target file to the receiving server. The transmission request carries the information digest value, so that the receiving server obtains the information digest value and initiates a dump request to the scheduling server. The scheduling server then matches a target storage server from multiple storage servers based on the information digest value to receive the target file dumped by the receiving server.

[0036] The transmission module is configured to sequentially transmit file fragments of the target file to the receiving server, so that when the receiving server receives the file fragments, it sequentially dumps the file fragments to the target storage server, and the target storage server combines all the file fragments to obtain the target file after all the file fragments of the target file have been transmitted.

[0037] According to a fourth aspect of this application, a file fragmentation transmission apparatus is provided, applied to a receiving server, the apparatus comprising:

[0038] The second request module is configured to send a dump request to the scheduling server after receiving a target file transmission request initiated by the client. The transmission request carries a message digest value of the target file. The message digest value is obtained based on the target byte count after determining the target byte count according to the target file size, so that the scheduling server can match a target storage server from multiple storage servers based on the message digest value to receive the target file dumped by the receiving server.

[0039] The dump module is configured to receive file fragments of the target file transmitted by the client and dump the file fragments to the target storage server, so that the target storage server can combine all file fragments to obtain the target file after all file fragments of the target file have been transmitted.

[0040] According to a fifth aspect of this application, an electronic device is provided, comprising: a memory and a processor;

[0041] The memory stores computer-executed instructions;

[0042] The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the file fragmentation transfer method, or another file fragmentation transfer method.

[0043] According to a sixth aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, are used to implement the file fragmentation transmission method described above, or another file fragmentation transmission method described above.

[0044] According to a seventh aspect of this application, a computer program product is provided, the computer program product including computer program code, which, when run on a computer, causes the computer to execute the file fragmentation transfer method, or another file fragmentation transfer method.

[0045] According to an eighth aspect of this application, a file fragmentation transmission system is provided, comprising a scheduling server, multiple storage servers, a client, and at least one receiving server; wherein,

[0046] The scheduling server is configured to, upon receiving a dump request initiated by the receiving server, match a target storage server from multiple storage servers based on a message digest value to receive the target file dumped by the receiving server.

[0047] The target storage server among the plurality of storage servers is used to receive file fragments of the target file transmitted by the corresponding receiving server, and combine all file fragments to obtain the target file;

[0048] The client is used to execute the file fragmentation transmission method described above; the receiving server is used to execute another file fragmentation transmission method described above.

[0049] In one implementation, the target storage server is specifically used to combine all file fragments to obtain the target file when the total value of the first digest information of the target file is consistent with the total value of the second digest information of all file fragments of the target file.

[0050] It is understood that the file fragmentation transmission method, apparatus, device, medium, product, and system provided in this application first determine the target number of bytes based on the file size of the target file on the client side, and obtain the message digest value corresponding to the target number of bytes. In the initial stage, the message digest value is transmitted to the receiving server. The receiving server initiates a dump request to the scheduling server. The scheduling server uses the message digest value to allocate the corresponding target dump server. This process replaces the method of allocating storage servers using the complete message digest value of the target file in related technologies. Instead, it uses a partial message digest value to complete the allocation of dump servers, which shortens the delay and blockage caused by the calculation of the message digest value due to the large file size in the early stage, improves the process startup performance, and reduces the overall process execution time. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1a This is one of the flowcharts illustrating a file fragmentation transmission method provided in an embodiment of this application;

[0053] Figure 1b A second schematic flowchart illustrating a file fragmentation transmission method provided in this application embodiment;

[0054] Figure 2 This is a flowchart illustrating step S101 in Figure 1;

[0055] Figure 3 A flowchart illustrating another file fragmentation transmission method provided in this application embodiment;

[0056] Figure 4 A flowchart illustrating yet another file fragmentation transmission method provided in this application embodiment;

[0057] Figure 5 A flowchart illustrating another file fragmentation transmission method provided in an embodiment of this application;

[0058] Figure 6A schematic diagram of a file fragmentation transmission device provided in an embodiment of this application;

[0059] Figure 7 A schematic diagram of another file fragmentation transmission device provided in the embodiments of this application;

[0060] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0061] Figure 9 This is a schematic diagram of a file fragmentation transmission system provided in an embodiment of this application.

[0062] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0063] During data transmission, problems such as slow large file transfer speeds are frequently encountered. For example, when uploading large files using browser download technology, issues such as slow upload speeds, file size limits imposed by gateways, interruptions during upload, and file incompleteness due to lost data packets in HTTP transmission are common. Related technologies propose solutions that involve segmenting and transmitting large files to ensure the integrity of files received by the server and to achieve efficient transmission.

[0064] During file upload, after one or more servers receive a large file locally, they usually need to dump it to a remote dedicated storage server. Since the storage capacity of each storage server is limited, it may be difficult to meet the dumping needs of the server when there are many files. In related technologies, multiple storage servers are scheduled and the corresponding target storage server is assigned to the file to realize the file dumping, thereby greatly improving the overall throughput and storage capacity of the system.

[0065] However, the dumping process can only be carried out after a corresponding target storage server is allocated to the file. Since the allocation of the target storage server in related technologies depends on the client calculating the complete message-digest algorithm 5 (MD5) value of the file, and for large files, especially those 4GB or larger, the calculation of the digest value requires a long time, the allocation time of the target storage server is relatively long. This results in a large delay and blockage in the initial stage of the file dumping process, which in turn affects the efficiency of file upload to a certain extent.

[0066] In view of the above, this application provides a file fragment transmission method, apparatus, device, medium, product, and system to address the aforementioned technical problems. The client determines the target byte count based on the target file size, obtains the information digest value of the target byte count, and initiates a transmission request for the target file to a receiving server. The transmission request carries the information digest value, enabling the receiving server to obtain the information digest value and initiate a dump request to a scheduling server. The scheduling server then matches a target storage server from multiple storage servers based on the information digest value to receive the target file dumped by the receiving server. File fragments of the target file are then transmitted sequentially to the receiving server. Upon receiving the file fragments, the receiving server sequentially dumps the file fragments to the target storage server, obtains the first total information digest value of the target file, and transmits the first total information digest value to the receiving server. Finally, in response to the completion of transmission of all file fragments of the target file, a verification instruction is sent to the receiving server to verify whether the first total information digest value and the second total information digest value of all file fragments are consistent. If they are consistent, the client triggers the combination of all file fragments in the target storage server to obtain the target file.

[0067] In the above process, the client first determines the target number of bytes based on the file size of the target file, then obtains the message digest value corresponding to that target number of bytes. The scheduling server uses this message digest value to allocate the corresponding dump server to the receiving server. This process replaces the method in related technologies that uses the complete message digest value of the target file to allocate the storage server, and instead uses a partial message digest value to complete the allocation of the dump server. This shortens the delay and congestion caused by the large file size in the initial stage when calculating the message digest value, improves the process startup performance, and reduces the overall process execution time. In addition, by obtaining the complete message digest value of the target file in parallel during file fragment transmission and performing the verification of the complete message digest value after the file fragment transmission is completed, the client further distributes the time requirement for calculating the complete message digest value of the file, thereby improving the file upload efficiency.

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0069] In one possible application scenario, a user needs to upload a target file (e.g., a 5GB file) from a terminal device to a receiving server, which then needs to dump the file. This application scenario includes a terminal device, a receiving server, a scheduling server, and a storage server. There can be multiple receiving and storage servers, and their numbers can be the same or different. The terminal device and all servers are interconnected via wired or wireless networks. Optionally, the terminal device receives the target file selected by the user, determines the message digest value of the target file, and initiates a transfer request to the receiving server. The receiving server, based on the data provided by the terminal device, initiates a dump request to the scheduling server. The scheduling server, based on the message digest value determined by the client, selects a target storage server from all storage servers for the receiving server to dump the target file.

[0070] The terminal devices may include, but are not limited to, computers, smartphones, tablets, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, portable computers, in-vehicle computers, wearable devices, desktop computers, set-top boxes, smart TVs, etc.

[0071] Each server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0072] It should be noted that the file fragmentation transmission method provided in this application can be used in the fintech field. It can also be used in any field other than fintech; the application field of the file fragmentation transmission method in this application is not limited.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0074] The application scenarios of this application have been briefly described above. The following will take the application to terminal devices and receiving servers as examples to explain in detail the file fragmentation transmission method provided by the embodiments of this application.

[0075] Please refer to Figure 1a , Figure 1a The flowchart of a file fragmentation transmission method provided in this application embodiment includes the following steps S101-S103.

[0076] Step S101: Determine the target number of bytes based on the target file size, and obtain the information digest value of the target number of bytes.

[0077] In related technologies, to achieve real-time dumping of target files from the receiving server to the storage server, a technical solution has been proposed that uses the MD5 value of the target file to allocate a storage server to the receiving server. In this process, typically, before file transfer occurs on the client (a terminal device with a browser installed), the MD5 value of the file is calculated and sent to the receiving server along with the transfer request. Upon receiving the file, the receiving server uses this MD5 value to allocate a target storage server.

[0078] However, in practical applications, MD5 calculation has a latency, especially for large files, such as those exceeding 4GB, and when multiple large file transfer requests exist simultaneously, resulting in a long MD5 calculation latency. This will cause significant delays and congestion in the early stages of file transfer. Therefore, in this embodiment, the client does not calculate the complete MD5 value of the target file before initiating a file transfer request. Instead, it determines the target byte count based on the size of the target file. Those skilled in the art can adapt the target byte count to the actual application. Generally, the target byte count is less than the total number of bytes in the target file. To ensure the MD5 calculation latency of the target file, the target byte count can be determined as the first 1024 bytes of the target file, and the MD5 value (MD5_1024) of the first 1024 bytes can be calculated. In some applications, the target byte count can also be determined as the first 512 bytes, the first 2048 bytes, or even the first 4096 bytes of the target file, depending on the size of the target file.

[0079] This embodiment calculates a partial message digest value corresponding to the target number of bytes instead of the full message digest value, and uses this partial message digest value to complete the allocation of the dump server. This effectively shortens the delay and blockage caused by the large file size in the initial stage, improves the process startup performance, and reduces the overall process execution time.

[0080] In one implementation, considering that some files have relatively small memory footprints and the latency of the MD5 calculation process can be negligible, in order to improve the efficiency of determining the target byte count, such as... Figure 2 As shown, step S101 of this embodiment, which determines the target number of bytes based on the target file size, may include the following steps:

[0081] Step S101a: When the target file size is greater than or equal to the preset number of bytes, the first N bytes of the target file are determined as the target number of bytes; where N is a positive integer.

[0082] Step S101b: When the target file size is less than the preset number of bytes, determine the total number of bytes in the target file as the target number of bytes.

[0083] It should be noted that those skilled in the art can adapt the preset number of bytes and the value of N to suit specific applications.

[0084] For example, the preset number of bytes and N can be set to the same number of bytes. For instance, the first N bytes can be set to the first 1024 bytes. When the target file size is less than 1024 bytes, the target file's bytes are directly used as the target number of bytes. In other words, when the target file size is less than the corresponding number of bytes, the calculation of its complete MD5 value is performed. The latency of this complete MD5 value calculation is negligible, and there is no need to further calculate its partial digest value. However, for target files larger than 100MB, calculating the partial MD5 value from the first 1024 bytes of the target file can significantly reduce the calculation latency.

[0085] In this embodiment, from a business logic perspective, the receiving server's selection of a storage server no longer depends on the complete file MD5 value, but rather weakens to depend on MD5_1024. Thus, when the upload process of the target file starts, during the directory allocation process in the scheduling server, the calculation of the directory MD5 can be optimized to a constant value A. This A is only related to the speed at which the browser calculates the (1024 bytes) MD5, and is no longer the Fmd5(size) function that grows linearly with the file size over time.

[0086] Step S102: Initiate a transmission request for the target file to the receiving server. The transmission request carries the information digest value, so that the receiving server can obtain the information digest value and initiate a dump request to the scheduling server. The scheduling server can then match a target storage server from multiple storage servers based on the information digest value to receive the target file dumped by the receiving server.

[0087] Since this embodiment only needs to calculate the MD5 value (MD5_1024) corresponding to the target number of bytes for large files, the client can initiate a transmission request to the receiving server more quickly after receiving the target file selected by the user. The receiving server initiates a dump request to the scheduling server, and the scheduling server allocates a target storage server for it based on the MD5 value of this part.

[0088] Optionally, the scheduling server can create a two-level directory structure (two characters per level) based on the possible combinations of the first four characters of the MD5 digest. The allocated directory structure might look something like aa / ab, 13 / ce. A directory structure table is created, and the directories in the table are associated with various storage servers. A corresponding number of directories are allocated to each storage server, which can be done in a direct proportionality between the storage space of the storage server and the number of directories. Upon receiving MD5_1024, the scheduling server matches the first four characters of MD5_102 with the directories in the target structure table and uses the association information to find the corresponding storage server as the target storage server, thus achieving rapid allocation of the target storage server.

[0089] In one implementation, to facilitate the orderly management of multiple target files by the receiving server, the transmission request in step S102 also carries the unique identifier of the target file and the size of the target file.

[0090] In one implementation, the user first selects the target file to be transferred and the transfer object (receiving server) on the terminal device (i.e., the client). There can be one or more target files and receiving servers. The client obtains the target file selected by the user, reads the size of the target file and calculates the MD5_1024 value of the target file. At the same time, it creates a unique TaskID for the corresponding target file and then sends a transfer request for the target file to the corresponding receiving server. The transfer request carries both the TaskID and the size of the target file.

[0091] When the receiving server receives the transmission request, it can generate task information based on the unique identifier of the target file, the size of the target file, and the message digest value, and store the task information in the target database. The receiving server can obtain the message digest value and initiate a dump request to the scheduling server. It can retrieve the task information from the target database, obtain the message digest value corresponding to the target file based on the task information, and initiate a dump request to the scheduling server.

[0092] Step S103: The file fragments of the target file are transmitted sequentially to the receiving server, so that when the receiving server receives the file fragments, it sequentially dumps the file fragments to the target storage server, and the target storage server combines all the file fragments to obtain the target file after all the file fragments of the target file have been transmitted.

[0093] In this embodiment, after the scheduling server assigns a target storage server for the target file to the receiving server, the client segments the target file. The number and size of the segments can be determined based on the size of the target file, which is existing technology and will not be elaborated upon in this embodiment. After segmentation, the file segments are transmitted to the receiving server sequentially. Upon receiving the file segments, in order to improve the performance of the receiving server, it does not store the file segments locally, but uses the Secure File Transfer Protocol (SSH File Transfer Protocol, SFTP) to transfer the file segments to the target storage server in real time.

[0094] Optionally, the receiving server can create a Secure File Transfer Protocol (SFTP) operation API interface and a corresponding SFTP protocol client, and establish an SFTP channel with the target storage server. This SFTP channel is used to interact with the target storage server via SFTP commands, enabling real-time transmission of file fragments. Furthermore, considering that SFTP does not support random access to files, only append access, and each fragment must be written to a specified offset in the stored file through random access, this embodiment can utilize SFTP's append mode, setting the absolute offset to a negative offset of offset - size, to indirectly achieve random access to files, thereby enabling random writing of file fragments. When all file fragments have been transmitted, the complete dump of the target file in the target storage server can be achieved.

[0095] In one implementation, to effectively ensure the integrity of fragmented transmission, this embodiment uses the complete MD5 value of the file to verify file fragmentation. Specifically, as follows: Figure 1b As shown, it also includes the following step S104.

[0096] Step S104: Obtain the total first message digest value of the target file and transmit the total first message digest value to the receiving server;

[0097] Step S105: In response to the completion of transmission of all file fragments of the target file, a verification instruction is sent to the receiving server;

[0098] Step S103, which involves the target storage server combining all file fragments of the target file to obtain the target file after all file fragments of the target file have been transferred, specifically includes the following steps:

[0099] When the receiving server receives the verification instruction, it verifies whether the total value of the first message digest and the total value of the second message digest of all file fragments are consistent. If they are consistent, it triggers the combination of all file fragments in the target storage server to obtain the target file.

[0100] Currently, during file transfer, regardless of whether the file is transferred as a whole or in fragments, MD5 verification is typically required to ensure file integrity. Before the client initiates file transfer, the MD5 value of the file is calculated and sent to the receiving server along with the transfer request. Upon receiving the file, the receiving server verifies both the MD5 value of the received file and the MD5 value of the original file. Unlike existing technologies, this embodiment calculates the MD5 value of the target file (i.e., the total first message digest value, or source MD5) during fragmented file transfer, reducing the latency of calculating the complete MD5 value initially.

[0101] In this embodiment, the total first message digest value of the target file can be obtained by calculating it using the MD5 algorithm. Since calculating the complete message digest value of the target file takes a relatively long time, this embodiment can initiate the calculation of the complete MD5 of the target file when the client executes the file fragment transmission process. The file fragment transmission process occurs after the client initiates a transmission request to the receiving server. In other words, the complete message digest value of the target file can be calculated in parallel while the client transmits file fragments of the target file to the receiving server. This way, when the file fragment transmission is complete, the complete message digest value and the total message digest value of the received file fragments can be verified.

[0102] It should be noted that this embodiment does not limit the order between steps S103 and S104. That is, the total value of the first information digest of the target file can be calculated when the client segments the target file, or when the first file segment of the target file is transmitted to the receiving server, or when a transmission request is initiated to the receiving server, or when the client transmits the nth file segment to the receiving server, or in any other order. This embodiment takes the transmission of file segments in steps S104 and S103 as an example of parallel execution.

[0103] Understandably, in practical applications, the time for file fragmentation is usually longer than the calculation time for the MD5 value. In one possible implementation, when the receiving server receives the total value of the first message digest, it can store it in the local database and wait for the parallel business to complete, that is, after all file fragments have been transmitted. In subsequent processes, the total value of the first message digest is retrieved from the local database for verification.

[0104] Understandably, the total second message digest value of all file fragments can be calculated by the target storage server when it receives each file fragment using fragment MD5. After the file fragments have been transmitted, the total second message digest value of the transmitted file fragments is calculated using all fragment MD5 values. If the total first message digest value and the total second message digest value are consistent, it means that all file fragments have been transmitted successfully, ensuring the integrity of the target file. The target storage server can then combine the file fragments to obtain the target file and complete the storage of the target file.

[0105] Optionally, to further ensure the effectiveness of file fragment transmission, this embodiment allows the client to resend the corresponding updated file fragments according to the update instructions from the receiving server during the file fragment transmission process. Specifically, as shown... Figure 3 As shown, after transmitting file fragments to the receiving server in step S103, this embodiment further includes the following step S301.

[0106] Step S301: If an update instruction initiated by the receiving server is received, the updated file fragment corresponding to the file fragment is resent to the receiving server; wherein, the update instruction is issued by the receiving server when verifying that the information digest value of the file fragment is inconsistent with the preset information digest value, and is used to instruct the resentment of the updated file fragment of the corresponding file fragment.

[0107] In this embodiment, the receiving server verifies each file fragment it receives, and if the verification fails, it re-acquires the updated file fragment by issuing an update instruction. After the verification passes, the file fragment is dumped to the target storage server, which can effectively reduce file transmission integrity issues.

[0108] In one implementation, the transmission request is in the form of an HTTP POST request, and the file fragmentation request is in the form of an HTTP PUT request.

[0109] Understandably, HTTP PUT requests are defined as idempotent methods, meaning that for the same request, the later request will overwrite the earlier one. In contrast, HTTP POST requests do not overwrite the first request, and for the same request, multiple files may be generated.

[0110] Combining the above-described update process for file fragments, the use of HTTP PUT requests allows for direct overwriting of fragments without creating multiple file fragments, effectively avoiding data redundancy within the same file fragment.

[0111] Please refer to Figure 4 , Figure 4This is a flowchart illustrating another file fragmentation transmission method provided in an embodiment of this application, applied to a receiving server. The method includes the following steps:

[0112] Step S401: After receiving the target file transmission request initiated by the client, a dump request is initiated to the scheduling server. The transmission request carries the information digest value of the target file. The information digest value is obtained based on the target byte number after determining the target byte number according to the target file size, so that the scheduling server matches a target storage server from multiple storage servers to receive the target file dumped by the receiving server based on the information digest value.

[0113] Step S402: Receive the file fragments of the target file transmitted by the client, and dump the file fragments to the target storage server, so that the target storage server can combine all the file fragments to obtain the target file after all the file fragments of the target file have been transmitted.

[0114] In one implementation, the method further includes: receiving a first message digest total value of the target file transmitted by the client;

[0115] Upon receiving a verification instruction, the system verifies whether the total value of the first message digest and the total value of the second message digest of all file fragments are consistent; wherein, the verification instruction is issued by the client in response to the completion of transmission of all file fragments of the target file;

[0116] The step of having the target storage server combine all file fragments to obtain the target file after all file fragments of the target file have been transmitted includes:

[0117] When the total value of the first message digest and the total value of the second message digest are the same, the combination of all file fragments in the target storage server is triggered to obtain the target file.

[0118] Optionally, the total value of the first message digest is obtained in parallel by the client when sending file fragments to the receiving server.

[0119] In one implementation, the transmission request also carries a unique identifier for the target file and the size of the target file;

[0120] After receiving the client's request to transfer the target file, and before initiating a dump request to the scheduling server, the method may further include the following steps:

[0121] Task information is generated based on the unique identifier of the target file, the size of the target file, and the message digest value, and the task information is stored in the target database;

[0122] The step of initiating a dump request to the scheduling server includes the following steps: retrieving the task information from the target database, obtaining the information digest value corresponding to the target file based on the task information, and initiating a dump request to the scheduling server.

[0123] In one embodiment, the method may further include the following steps:

[0124] The message digest value of the file fragment is checked to see if it is consistent with the preset message digest value. If they are inconsistent, an update instruction is sent to the client. The update instruction is used to instruct the client to resend the updated file fragment of the corresponding file fragment.

[0125] It should be noted that the above solution provided in this embodiment is the interaction process of the client in the above embodiment, which can correspondingly implement all the method steps implemented in the above embodiment and achieve the same technical effect. Here, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail.

[0126] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating another file fragmentation transmission method provided in this application embodiment. This embodiment is described as a whole in conjunction with the client, receiving server, scheduling server, and storage server, including the following process:

[0127] Step S501: The user selects the target file on the client.

[0128] Step S502: If the user does not select a target file, the client can use a JS script to obtain the target file selected by the user through the onchange event.

[0129] Step S503: The client reads the target file size FILE_SIZE.

[0130] Step S504: The client determines the target file size in bytes and calculates the message digest value corresponding to the target file size, for example, calculating the target file MD5_1024. If the target file size is less than 1024 bytes, the full-text MD5 is calculated, which is applicable to files larger than 100M.

[0131] Step S505: The client creates a unique string TaskID for the target file.

[0132] Step S506: The client sends the upload / begin request (file transfer request, carrying basic information of the target file: MD5_1024, TaskID, FILE_SIZE) from Step 1 to the receiving server in the form of an HTTP POST request.

[0133] Step S507: The receiving server generates corresponding task information based on the basic information of the target file, stores it in the target database, and sends a dump request to the scheduling server.

[0134] Step S508: The storage server initiates a server registration request to the scheduling server, which can be done by submitting a form.

[0135] Step S509: The scheduling server receives the registration request, registers the storage server, and records each registered storage server.

[0136] Step S510: Create a directory structure table for MD5, which contains 256*256 directories formed by possible combinations of the first four digits of the MD5 digest.

[0137] Step S511: Associate the directories in the target structure table with the registered storage servers to obtain configuration information.

[0138] Understandably, the above steps S501-S511 represent the initial stage of file fragment transmission.

[0139] It should be noted that there is no specific order between steps S508-S511 and steps S501-S507. The storage of the storage server and the allocation of the directory structure table can be performed before or after the selection of the target file. This embodiment takes S508 as the first step.

[0140] Step S512: The scheduling server receives the dump request and matches the target storage server for the target file based on the first 4 characters of the target file's information digest value and the configuration information.

[0141] Step S513: The receiving server obtains and records the information of the target storage server matched by the scheduling server.

[0142] Step S514: The client performs a fragmentation operation on the target file and initializes the fragments, for example, setting the total number of fragments to 'a' and the initial fragment number to 0, to facilitate recording the fragmentation transmission process. Simultaneously, step S526 is executed in parallel.

[0143] Step S515: Client sets execution conditions: Determine if there are any unsent fragments. If there are, proceed to step S516; otherwise, proceed to step S527.

[0144] Step S516: The client calculates the file fragment MD5 and file fragment SIZE, etc.

[0145] Step S517: The client sends the upload / slice request (file slice transfer request, carrying the MD5 of the file slice source) from step 2 to the receiving server in the form of an HTTP PUT request.

[0146] Specifically, the client can use HTTP PUT requests to repeatedly upload fragmented data to the receiving server after the fragments have been divided.

[0147] Step S518: The receiving server performs MD5 verification on the file fragments to verify whether the MD5 of the received file fragments is consistent with the MD5 of the file fragment source.

[0148] Step S519: After the MD5 value verification of the file fragment passes, the receiving server locks the current (storage) file creation (operation) permissions through a global lock control (e.g., redission-rlock).

[0149] Step S520: The target storage server confirms the file creation status.

[0150] Step S521: The receiving server determines whether the current file has been created based on the file creation status of the target storage server. If the file has not been created, the receiving server instructs the target storage server to create the file. If the file has been created, step S522 is executed.

[0151] Specifically, the file can call the SFTP write method with a specified offset to create a file of a specified size, and the storage file size is determined based on the size of the passed-in file.

[0152] Step S522: The target storage server creates the file according to the instructions of the receiving server.

[0153] Step S523: Unlock the creation permission of the current file and lock the write (write operation) permission of the current file fragment.

[0154] Locking file creation permissions means locking the global file creation permission for the MD5 digest value of the source file; unlocking file creation operation permissions means unlocking the global file creation permission for the MD5 digest value of the source file.

[0155] Step S524: Under the write permission of the file fragment, the target storage server writes the file fragment into the storage file.

[0156] Step S525: Unlock write permissions for the current file segment to facilitate writing to subsequent file segments.

[0157] Understandably, steps S505-S525 above are the fragmentation transmission and dumping stages.

[0158] Step S526: The client calculates the complete MD5 value of the target file.

[0159] Step S527: The client sends a file MD5 setting request (upload / md5) to the receiving server to synchronize the complete MD5 value of the target file. The receiving server receives the complete MD5 value of the target file and sets the complete MD5 value of this upload task in the server database. The request method can be a POST request.

[0160] In this embodiment, a parallel process for the client to calculate the complete MD5 of a file is created. This process is synchronized with the file chunk upload after the client responds to the server to create the file (upload / begin). The complete MD5 value of the file is calculated and sent to the receiving server.

[0161] Understandably, steps S526 and S527 above constitute the complete MD5 synchronization phase of the file.

[0162] Step S528: Wait at the parallel business aggregation node for all preceding parallel processes (file fragmentation and transmission completed, MD5 calculation completed) to be completed.

[0163] The client's parallel process aggregation node waits for the file slice upload and file MD5 setting to be completed by setting a cyclic delay waiting mechanism on the client side, and then enters the end stage of the process.

[0164] Step S529: The client sends a request to the receiving server for Step 3: upload / finish request (file fragment transfer completion request).

[0165] Understandably, when both independent parallel processes have their end flags set, the client triggers the termination of the process.

[0166] Step S530: The receiving server sends a message to the storage server to calculate the total MD5 hash of the file fragments. At the same time, it retrieves the complete MD5 hash of the target file sent by the client from the local database, verifies whether they are consistent, and returns the result of successful or failed transmission of the target file to the client.

[0167] Step S531: Process terminated.

[0168] This embodiment employs a parallel business mechanism and a partial MD5 mechanism to optimize the Fmd5(size) delay, which increases linearly with the file size in the initial stage, into a constant A with a fixed minimum delay. This significantly improves the startup and response speed of the entire business process during file transfer, effectively increases transmission efficiency, and enhances the user experience.

[0169] It should be noted that the above solution provided in this embodiment includes the interaction process between the client and the receiving server in the above embodiments, which can correspondingly implement all the method steps implemented in the above embodiments and achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0170] This application also provides a file fragmentation transmission device for use on a client side, such as... Figure 6 As shown, the device includes:

[0171] The acquisition module 61 is configured to determine the target number of bytes based on the target file size and acquire the information digest value of the target number of bytes;

[0172] The first request module 62 is configured to initiate a transmission request for the target file to the receiving server. The transmission request carries the information digest value, so that the receiving server can obtain the information digest value and initiate a dump request to the scheduling server, and the scheduling server can match a target storage server for receiving the target file dumped by the receiving server from multiple storage servers according to the information digest value.

[0173] The transmission module 63 is configured to sequentially transmit file fragments of the target file to the receiving server, so that when the receiving server receives the file fragments, it sequentially dumps the file fragments to the target storage server, and the target storage server combines all the file fragments to obtain the target file after all the file fragments of the target file have been transmitted.

[0174] It should be noted that the apparatus provided in this application can implement all the method steps implemented by the client in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0175] This application also provides a file fragmentation transmission device, applied to a receiving server, such as... Figure 7 As shown, the device includes:

[0176] The second request module request 71 is configured to send a dump request to the scheduling server after receiving a target file transmission request initiated by the client. The transmission request carries an information digest value of the target file. The information digest value is obtained based on the target byte count after determining the target byte count according to the target file size, so that the scheduling server matches a target storage server from multiple storage servers based on the information digest value to receive the target file dumped by the receiving server.

[0177] The dump module 72 is configured to receive file fragments of the target file transmitted by the client and dump the file fragments to the target storage server, so that the target storage server can combine all file fragments to obtain the target file after all file fragments of the target file have been transmitted.

[0178] It should be noted that the apparatus provided in this application can implement all the method steps implemented by the receiving server in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0179] This application also provides an electronic device, such as... Figure 8 As shown, it includes: a transceiver 81, a memory 83, and a processor 82;

[0180] The memory 83 stores computer-executed instructions;

[0181] The processor 82 executes computer execution instructions stored in the memory, causing the electronic device to perform the file fragmentation transmission method, or another file fragmentation transmission method.

[0182] The processor 82 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0183] The memory 83 is connected to the processor 82 via the system bus and completes communication between them. The memory 83 is used to store computer program instructions.

[0184] Transceiver 81 can be used to obtain the task to be run and its configuration information.

[0185] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0186] It should be noted that the electronic device provided in this application can implement all the method steps implemented by the client or receiving server in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0187] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the file fragmentation transmission method or another file fragmentation transmission method.

[0188] It should be noted that the computer-readable storage medium provided in this application can implement all the method steps implemented by the client or receiving server in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0189] This application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute the file fragmentation transmission method or another file fragmentation transmission method.

[0190] It should be noted that the program product provided in this application can implement all the method steps implemented by the client or receiving server in the above method embodiments and achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.

[0191] This application also provides a file fragmentation transmission system, such as... Figure 9As shown, it includes a scheduling server 91, multiple storage servers 92, a client 93, and at least one receiving server 94; wherein,

[0192] The scheduling server 91 is used to match a target storage server for receiving the target file dumped by the receiving server 94 from multiple storage servers based on the information digest value when it receives a dump request initiated by the receiving server 94.

[0193] The target storage server among the plurality of storage servers 92 is used to receive file fragments of the target file transmitted by the corresponding receiving server 94, and combine all file fragments to obtain the target file;

[0194] The client 93 is used to execute a file fragmentation transmission method provided in the method embodiment; the receiving server 94 is used to execute another file fragmentation transmission method provided in the method embodiment.

[0195] In one implementation, the target storage server is specifically used to combine all file fragments to obtain the target file when the total value of the first digest information of the target file is consistent with the total value of the second digest information of all file fragments of the target file.

[0196] It should be noted that the system provided in this application can implement all the method steps implemented by the client and the receiving server in the above method embodiments, and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0198] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0199] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0200] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0201] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0202] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0203] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0204] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0205] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0206] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0207] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0208] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A file fragment transmission method, characterized by, The method applied to a client comprises: determining a target byte number according to a target file size, and obtaining an information digest value of the target byte number; when the target file size is greater than a preset byte number, determining a first N byte number of the target file as the target byte number; wherein N is a positive integer, and N is the preset byte number; initiating a transmission request of the target file to a receiving server, the transmission request carrying the information digest value, so that the receiving server obtains the information digest value and initiates a dump request to a scheduling server, and the scheduling server matches a target storage server for receiving the target file dumped by the receiving server from a plurality of storage servers according to the information digest value; transmitting file fragments of the target file to the receiving server in sequence, so that the receiving server sequentially dumps the file fragments to the target storage server when the file fragments are received, and the target storage server combines all the file fragments to obtain the target file when all the file fragments of the target file are transmitted; further comprising: obtaining a first information digest total value of the target file, and transmitting the first information digest total value to the receiving server; and sending a check instruction to the receiving server in response to the transmission of all the file fragments of the target file being completed; the target storage server combines all the file fragments to obtain the target file when all the file fragments of the target file are transmitted, comprising: when the receiving server receives the check instruction, checking whether the first information digest total value and a second information digest total value of all the file fragments are consistent, and if so, triggering the target storage server to combine all the file fragments to obtain the target file; wherein the first information digest total value is a complete information digest value of the target file, and the client calculates the first information digest total value of the target file in parallel when transmitting the file fragments of the target file to the receiving server.

2. The method of claim 1, wherein, the target byte number is determined according to the target file size, comprising: when the target file size is equal to the preset byte number, the first N byte number of the target file is determined as the target byte number; wherein N is a positive integer; when the target file size is less than the preset byte number, all the byte numbers of the target file are determined as the target byte number.

3. The method according to claim 1 or 2, characterized in that, The transmission request further carries a unique identifier of the target file and the target file size.

4. The method of claim 1, wherein, after transmitting the file fragments to the receiving server in sequence, further comprising: if an update instruction initiated by the receiving server is received, the updated file fragment corresponding to the file fragment is re-sent to the receiving server; wherein the update instruction is issued when the receiving server checks that the information digest value of the file fragment is inconsistent with a preset information digest value, and is used to indicate that the updated file fragment corresponding to the file fragment is re-sent.

5. A file fragment transmission method characterized by, The method applied to a receiving server comprises: After receiving a transmission request of a target file initiated by a client, a dump request is initiated to a scheduling server, wherein the transmission request carries an information digest value of the target file, the information digest value is obtained based on a target byte number determined according to a target file size, so that the scheduling server matches a target storage server for receiving the target file dumped by the receiving server according to the information digest value from a plurality of storage servers; when the target file size is greater than a preset byte number, the first N bytes of the target file are determined as the target byte number; wherein N is a positive integer, and N is the preset byte number; The file fragments of the target file transmitted by the client are received, and the file fragments are dumped into the target storage server, so that the target storage server combines all the file fragments to obtain the target file when all the file fragments of the target file are transmitted; Further comprising: The first information digest total value of the target file transmitted by the client is received; wherein the first information digest total value is a complete information digest value of the target file, and the client calculates the first information digest total value of the target file in parallel when transmitting the file fragments of the target file to the receiving server; When receiving a check instruction, check whether the first information digest total value and the second information digest total value of all the file fragments are consistent; wherein the check instruction is issued by the client in response to the transmission of all the file fragments of the target file being completed; The target storage server combines all the file fragments to obtain the target file when all the file fragments of the target file are transmitted, comprising: When the first information digest total value and the second information digest total value are consistent, trigger the target storage server to combine all the file fragments to obtain the target file.

6. The method of claim 5, wherein, The transmission request further carries a unique identifier of the target file and the target file size; Then, after receiving the transmission request of the target file initiated by the client, and before initiating the dump request to the scheduling server, the method further comprises: Generating task information according to the unique identifier of the target file, the target file size and the information digest value, and storing the task information into a target database; The dump request initiated to the scheduling server comprises: calling the task information from the target database, and obtaining the information digest value corresponding to the target file according to the task information, and initiating the dump request to the scheduling server.

7. The method according to claim 5 or 6, characterized in that, Further comprising: Check whether the information digest value of the file fragment is consistent with a preset information digest value, if not, send an update instruction to the client, the update instruction is used to indicate to resend the updated file fragment corresponding to the file fragment.

8. A file fragment transmission apparatus characterized by comprising: Applied to a client, the device comprises: The obtaining module is configured to determine a target byte number according to a target file size, and obtain an information digest value of the target byte number; when the target file size is greater than a preset byte number, the first N bytes of the target file are determined as the target byte number; wherein N is a positive integer, and N is the preset byte number; The first request module is configured to initiate a transmission request of the target file to a receiving server, the transmission request carrying the information digest value, so that the receiving server obtains the information digest value and initiates a dump request to a scheduling server, and the scheduling server matches a target storage server for receiving the target file dumped by the receiving server from a plurality of storage servers according to the information digest value; The transmission module is configured to sequentially transmit file fragments of the target file to the receiving server, so that the receiving server sequentially dumps the file fragments to the target storage server when the file fragments are received, and the target storage server combines all the file fragments to obtain the target file when all the file fragments of the target file are transmitted; The file fragment transmission device is also used to obtain a first information digest total value of the target file; wherein the first information digest total value is a complete information digest value of the target file, and the client calculates the first information digest total value of the target file in parallel when transmitting the file fragments of the target file to the receiving server; the first information digest total value is transmitted to the receiving server, and in response to the transmission of all the file fragments of the target file being completed, a check instruction is sent to the receiving server, so that when the receiving server receives the check instruction, it checks whether the first information digest total value and a second information digest total value of all the file fragments are consistent, and if they are consistent, triggers the target storage server to combine all the file fragments to obtain the target file.

9. A file fragment transmission apparatus characterized by comprising: The device applied to a receiving server comprises: The second request module is configured to initiate a dump request to a scheduling server after receiving a transmission request of a target file initiated by a client, wherein the transmission request carries an information digest value of the target file, the information digest value being obtained based on a target byte number determined according to a size of the target file, so that the scheduling server matches a target storage server for receiving the target file dumped by the receiving server from a plurality of storage servers according to the information digest value; when the size of the target file is greater than a preset byte number, the first N byte numbers of the target file are determined as the target byte number; wherein N is a positive integer, and N is the preset byte number; The dump module is configured to receive the file fragments of the target file transmitted by the client and dump the file fragments into the target storage server, so that the target storage server combines all the file fragments to obtain the target file when all the file fragments of the target file are transmitted. The file fragment transmission device is further configured to receive a first information digest total value of the target file transmitted by the client; wherein the first information digest total value is a complete information digest value of the target file, and the client calculates the first information digest total value of the target file in parallel when transmitting the file fragments of the target file to the receiving server; when receiving a check instruction, check whether the first information digest total value is consistent with a second information digest total value of all file fragments; wherein the check instruction is sent by the client in response to the transmission of all file fragments of the target file being completed, so that the target storage server combines all file fragments to obtain the target file when the first information digest total value is consistent with the second information digest total value.

10. An electronic device, comprising: Comprise: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the electronic device executes the file fragment transmission method in any one of claims 1-4, or the file fragment transmission method in any one of claims 5-7.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the file fragment transmission method in any one of claims 1-4, or the file fragment transmission method in any one of claims 5-7.

12. A computer program product, characterised in that, The computer program product comprises computer program code, which, when executed on a computer, causes the computer to execute the file fragment transmission method in any one of claims 1-4, or the file fragment transmission method in any one of claims 5-7.

13. A file fragment transmission system characterized by, Comprise a scheduling server, a plurality of storage servers, a client and at least one receiving server; wherein the scheduling server is configured to match a target storage server for receiving a target file dumped by the receiving server from a plurality of storage servers according to an information digest value when receiving a dump request initiated by the receiving server; the target storage server in the plurality of storage servers is configured to receive file fragments of the target file transmitted by the corresponding receiving server, and combine all file fragments to obtain the target file; the client is configured to execute the file fragment transmission method in any one of claims 1-4; the receiving server is configured to execute the file fragment transmission method in any one of claims 5-7.

14. The system of claim 13, wherein, The target storage server is specifically configured to combine all file fragments to obtain the target file when the first information digest total value of the target file is consistent with the second information digest total value of all file fragments of the target file.

Images