A File Upload System and Method Based on HTTP Protocol with Separation of Uplink and Downlink Paths

Abstract

This application provides a file upload system and method based on HTTP protocol with separate uplink and downlink connections. The system obtains a file list from the client and establishes an uplink HTTP link and a downlink feedback link. Each file in the file list is blindly sent unidirectionally to the server, and the unique file ID of each sent file is added to a pending confirmation set. After the server receives and writes the uploaded file, it sends the file processing result (including the file's unique ID and file reception status, such as success or failure) back to the client via the downlink feedback link. The client then receives the file processing result from the server and removes the file ID with a successful reception status from the pending confirmation set. By separating the upload link and the feedback link, the upload link can continuously send files without waiting for server feedback; the receiving link processes the server's feedback results in real time, making full use of bandwidth resources and improving upload efficiency.

Description

Technical Field

[0001] This application relates to the field of file transfer technology, and more specifically, to a file upload system and method based on HTTP protocol with separate uplink and downlink connections. Background Technology

[0002] In traditional HTTP upload systems, due to the limitations of the HTTP protocol, after sending one file, the system waits for the server to report a successful upload status before sending the next. If a large number of small files need to be uploaded, such as when home or business users use NAS (Network Attached Storage), which typically involves a large number of photos, music, videos, and other small files, the waiting time for file uploads will far exceed the data transfer time. This is because for a large number of small files, network transmission time is extremely short, while network round-trip time (RTT) and server processing time are relatively fixed, resulting in a significant proportion of the total upload time being spent waiting for feedback, leading to extremely low bandwidth utilization.

[0003] The common solution to this problem is to start multiple connections simultaneously (using multi-threading / multi-connection concurrency) in the hope of improving bandwidth utilization and transmission efficiency. However, this can only alleviate the problem to a limited extent and will significantly increase the server's connection maintenance overhead (TCP handshake, SSL handshake, memory usage). Furthermore, under high concurrency, it is easy to trigger the server's rate limiting policy, which can only alleviate the problem to a limited extent and cannot solve the serial blocking problem. Summary of the Invention

[0004] The purpose of this application is to provide a file upload system and method based on the HTTP protocol with separate uplink and downlink connections. By separating the upload link and the feedback link, the upload link can continuously send files without waiting for server feedback; the receiving link processes the server's feedback results on the files in real time, making full use of bandwidth resources and improving upload efficiency.

[0005] In a first aspect, embodiments of this application provide a file upload system based on the HTTP protocol with separate uplink and downlink connections, comprising: a client, configured to obtain a file list, establish an uplink HTTP link and a downlink feedback link, blindly send each file in the file list to the server in a one-way manner, and include the unique file number of the sent file in a set to be confirmed, wherein the file list contains multiple files to be uploaded, each file corresponding to a unique file number, and one-way blind sending means that the sender completes the previous file sending task and proceeds to the next file sending task without waiting for the receiver's response; a server, configured to receive the file uploaded by the client, and after writing, send the file processing result back to the client through the downlink feedback link, wherein the file processing result includes the unique file number and the file receiving status, the file receiving status being either successful or failed; the client is also configured to receive the file processing result fed back by the server, and based on the file processing result, delete the unique file number with a successful file receiving status from the set to be confirmed.

[0006] In conjunction with the first aspect, in the first possible implementation of the first aspect, the client is specifically used to: traverse the file list, generate a unique file ID (UUID) for each file to be uploaded, and construct a multi-part data packet. The multi-part data packet contains several multi-part sub-blocks, each corresponding to a file to be uploaded. Each multi-part sub-block contains a header and a body. The header includes the corresponding file's UUID, File_Size, and File_Name. The UUID is the unique file ID, File_Size is the file size, and File_Name is the file name. The client then retrieves the multi-part data packet from the queue, writes it to the Socket buffer of the uplink HTTP connection, and after the sending action is completed, adds all the UUIDs contained in the multi-part data packet to the pending confirmation set, immediately enters the next loop, and sends the next multi-part data packet.

[0007] In conjunction with the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the client is specifically used to: generate a multi-part data packet for each file to be uploaded in the file list, each multi-part data packet containing a multi-part sub-block; or, based on each file to be uploaded in the file list, adopt a dynamic packaging strategy to generate several multi-part data packets, each multi-part data packet containing at least one multi-part sub-block.

[0008] In conjunction with the second possible implementation of the first aspect, in the third possible implementation of the first aspect, when adopting a dynamic packaging strategy, the client is specifically used to: obtain the set upper limit of single packet size; obtain the number of UUIDs in the current set to be confirmed, and determine the upper limit of single packet quantity based on the number of UUIDs; determine the packaging files for this time from the file list in order based on the upper limit of single packet size and the upper limit of single packet quantity; and package the packaging files for this time to obtain the multi-part data packet for this packaging.

[0009] In conjunction with the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, the client is specifically configured to: when the number of UUIDs in the current set to be confirmed is higher than a first threshold, if the previous single packet quantity limit is greater than a predetermined minimum value, reduce the set quantity based on the previous single packet quantity limit; if the previous single packet quantity limit is not greater than the predetermined minimum value, determine the predetermined minimum value as the current single packet quantity limit; when the number of UUIDs in the current set to be confirmed is lower than a second threshold, obtain the number of UUIDs from the previous n times; if the number of UUIDs from the previous n times is lower than the second threshold, and the previous single packet quantity limit is less than a predetermined maximum value, reduce the set quantity based on the previous single packet quantity limit. Based on the above, an additional set quantity is added. If the number of UUIDs in the previous n times is all lower than the second threshold, and the previous single packet quantity limit is not less than the agreed maximum value, the agreed maximum value is determined as the single packet quantity limit for this time. If the number of UUIDs in the previous n times is not all lower than the second threshold, the previous single packet quantity limit is determined as the single packet quantity limit for this time. When the number of UUIDs in the current set to be confirmed is not lower than the second threshold and not higher than the first threshold, the agreed basic value is determined as the single packet quantity limit for this time. Among these, the agreed minimum value < the agreed basic value < the agreed maximum value, and the set quantity does not exceed the difference between the agreed minimum value and the agreed basic value and the difference between the agreed basic value and the agreed maximum value.

[0010] In conjunction with the first possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the server is specifically used for: listening to the uplink HTTP connection port and reading the data stream; parsing the multi-part data packet according to the multi-part specification, determining the UUID, File_Size, and File_Name corresponding to each multi-part sub-block in the multi-part data packet, and writing the data to a file; after writing, forwarding the file processing result {UUID, Success / Fail} to the internal message queue MQ to complete the parsing of this multi-part data packet and continuing to parse the next multi-part data packet, where UUID is the unique file number, Success / Fail is the file receiving status, Success indicates that the file upload was successful, and Fail indicates that the file upload failed; receiving the file processing result in the internal message queue MQ and feeding it back to the client through the downlink feedback link according to the SSE protocol.

[0011] In conjunction with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the client is specifically used to: listen to the downlink feedback link port, receive the file processing result from the server; determine the unique file number and file receiving status from the file processing result, and determine the unique file number whose file receiving status is successful; delete the unique file number whose file receiving status is successful from the set to be confirmed.

[0012] In conjunction with the first aspect, in the seventh possible implementation of the first aspect, the client is further configured to: after completing the traversal of the file list and setting the interval, obtain the UUID in the current set to be confirmed; determine the retransmitted file corresponding to the UUID in the current set to be confirmed from the file list, and clear the set to be confirmed; blindly send each retransmitted file in the file list to the server in one direction via the uplink HTTP connection, and include the unique file number of the sent retransmitted file in the set to be confirmed.

[0013] Secondly, embodiments of this application provide a file upload method based on HTTP protocol with separate uplink and downlink connections, applied to a client in a file upload system based on HTTP protocol with separate uplink and downlink connections, as described in any one of the first aspects or possible implementations of the first aspect. The method includes: obtaining a file list, wherein the file list contains multiple files to be uploaded, each file corresponding to a unique file number; establishing an uplink HTTP link and a downlink feedback link; unidirectionally sending each file in the file list to the server, and including the unique file number of the sent file in a set to be confirmed, wherein unidirectional blind sending means that the sender performs the next file sending task without waiting for the receiver's response after completing the previous file sending task; receiving the file processing result fed back by the server, and based on the file processing result, deleting the unique file number whose file receiving status is successful from the set to be confirmed, wherein the file processing result includes the unique file number and the file receiving status, and the file receiving status is success or failure.

[0014] Thirdly, embodiments of this application provide a file upload method based on HTTP protocol with separate uplink and downlink connections. The method is characterized by being applied to a server in a file upload system based on HTTP protocol with separate uplink and downlink connections, as described in any one of the first aspects or possible implementations of the first aspect. The method includes: receiving files uploaded by a client, wherein the client blindly sends each file in a file list to the server via an uplink HTTP link; and after writing is completed, feeding back the file processing result to the client via a downlink feedback link, wherein the file processing result includes a unique file number and a file receiving status, where the file receiving status is success or failure.

[0015] Beneficial Effects: This solution obtains a file list (containing multiple files to be uploaded, each with a unique file ID) through the client, establishes an uplink HTTP link and a downlink feedback link, and blindly sends each file in the file list to the server one-way, adding the unique file IDs of the sent files to the pending confirmation set. The server receives the files uploaded by the client, and after writing, sends the file processing result (including the file ID and file reception status) back to the client via the downlink feedback link. The client then receives the file processing result from the server and, based on the result, removes the unique file IDs with a successful reception status from the pending confirmation set. This solution breaks the native HTTP protocol's strongly bound synchronous "request-response" model, separating the upload of binary file data and the feedback of upload result status into two independent communication channels (which can be two TCP connections, or two independent logical streams on one connection). This transforms the uplink channel into a "one-way blind transmission" mode, completely eliminating the RTT (round-trip time) window caused by waiting for ACK packets. Meanwhile, this solution changes the client's sending logic from "send-block-receive-judgment-resend" to "loop read-continuous send-update local status table." The client's sending thread does not perform blocking read operations at all, relying solely on the underlying TCP protocol, thus maximizing bandwidth utilization during the sending phase. Furthermore, by using a unique file number in conjunction with a set of pending confirmations, the server's file reception status can be verified, ensuring the reliability and completeness of file uploads (verifying which files the server has already received, avoiding upload omissions due to various transmission failures).

[0016] During file uploads on the client side, a single-file packaging approach can be used. However, to reduce connection overhead and improve throughput, a dynamic packaging strategy is designed. In one-way blind transmission, the actual server load is ignored, and sending large batches of files without considering server capacity can lead to a significant increase in transmission failure rate. Therefore, this solution, when packaging files in batches, determines the appropriate upper limit for the current single-file quantity by comprehensively considering the number of UUIDs in the current pending confirmation set (which reflects server load to some extent), the number of UUIDs in the previous pending confirmation set, the agreed minimum value, the agreed maximum value, the number of UUIDs in the previous n pending confirmation sets, and the previous single-packet quantity limit. This dynamic adjustment strategy considers server load, determines an appropriate number of files to package, and avoids the increased transmission failure rate and surge in server load caused by blindly increasing the number of files per package. Furthermore, the single-packet quantity adjustment mechanism is relatively smooth, effectively preventing network instability.

[0017] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 An interaction diagram of a file upload system based on the HTTP protocol with separate uplink and downlink links provided in an embodiment of this application.

[0020] Figure 2 A flowchart of a file upload method based on the HTTP protocol with separate uplink and downlink operations, running on the client side.

[0021] Figure 3 A flowchart for a file upload method based on the HTTP protocol with separate uplink and downlink connections that runs on a server.

[0022] Figure 4 This is a diagram illustrating the batch uploading of small files. Detailed Implementation

[0023] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0024] When home or business users (mainly home users) use a NAS, they typically handle a large number of files, including photos, music, and videos (especially photos). Since it's the first time using the NAS, there will be a considerable number of small files (tens of thousands of photos are not uncommon) that need to be uploaded to the server. The system usually schedules uploads to occur at midnight the following day to avoid peak hours.

[0025] Please see Figure 1 , Figure 1 This is an interaction diagram of a file upload system based on the HTTP protocol with separate uplink and downlink connections, provided in an embodiment of this application. The file upload system mainly includes clients and servers; there can be multiple clients and multiple servers.

[0026] The client and server can work together to run a file upload method based on the HTTP protocol that separates the uplink and downlink, such as... Figure 2 and Figure 3As shown, the part running on the client may include steps S11, S12, S13, and S14, while the part running on the server may include steps S21 and S22.

[0027] During the file sending phase, the client is mainly used to obtain the file list, establish an uplink HTTP link and a downlink feedback link, send each file in the file list to the server in a one-way blind manner, and include the unique file number of the sent file in the set to be confirmed. The file list contains multiple files to be uploaded, each file corresponding to a unique file number. One-way blind sending means that the sender does not wait for the receiver's response after completing the previous file sending task.

[0028] When a file needs to be uploaded, the client can run step S11.

[0029] Step S11: Obtain the file list, which contains multiple files to be uploaded, each file having a unique file number.

[0030] In this embodiment, the client can obtain a file list containing multiple files to be uploaded. The client iterates through the file list and generates a unique file ID (UUID) for each file to be uploaded. Therefore, each file in the file list can correspond to a unique file ID (i.e., a UUID).

[0031] Afterwards, the client can run step S12.

[0032] Step S12: Establish the upstream HTTP link and the downstream feedback link.

[0033] In this embodiment, in order to separate sending and feedback, the client can establish an uplink HTTP link and a downlink feedback link. The uplink HTTP link is used for the client to send files to the server in one direction blindly, while the downlink feedback link is used for the server to send the file processing results back to the client, forming a "dual-channel asynchronous pipeline" architecture.

[0034] Uplink HTTP links can be one or more long links used to continuously push binary streams of files to the server (e.g., using HTTP multi-part uploads) without waiting for any immediate response.

[0035] The downlink feedback link can be an independent receiving connection, or it can be a WebSocket / SSE long polling to listen in real time to the file processing results returned by the server (e.g., file unique number + file receiving status, such as {UUID, Success / Fail}, or file unique number + file receiving status + error code, such as {UUID, Success / Fail, Reason}; this embodiment takes file unique number + file receiving status as an example).

[0036] A correlation mechanism can be established using the file's unique identifier (UUID): each file carries a globally unique UUID when it is sent. After the server completes the processing, it returns the UUID and the file processing status (Success / Fail) in the downlink channel (downlink feedback link). The client can then use this UUID to match the returned file's unique identifier with the file's unique identifier that was sent to verify the file's upload status.

[0037] After the upstream HTTP link and downstream feedback link are established, the client can proceed to step S13.

[0038] Step S13: Send each file in the file list to the server in a one-way blind manner, and add the unique file number of the sent file to the set to be confirmed. Here, one-way blind sending means that the sender does not wait for the receiver's response after completing the previous file sending task.

[0039] In this embodiment, for the file to be sent, the client can construct a multi-part data packet, wherein the multi-part data packet contains several multi-part sub-blocks (at least one multi-part sub-block), and each multi-part sub-block corresponds to a file to be uploaded. The multi-part sub-block contains a data header and a data body. The data header includes the UUID, File_Size, and File_Name of the corresponding file. The UUID is the unique number of the corresponding file, the File_Size is the size of the corresponding file, and the File_Name is the name of the corresponding file.

[0040] One way to construct a multi-part data packet is to generate a multi-part data packet for each file to be uploaded in the file list according to the order of the files to be uploaded in the file list. In this way, each multi-part data packet contains a multi-part sub-block.

[0041] To reduce connection overhead and improve throughput, a dynamic packetization strategy can also be adopted. In one-way blind transmission, the actual server load doesn't need to be considered during the transmission process; it simply sends data. However, blindly transmitting large batches of files in bulk makes it difficult to account for server load, easily leading to a significant increase in transmission failure rate. Therefore, this solution considers factors such as the number of UUIDs in the current pending confirmation set, the number of UUIDs in the previous pending confirmation set, the agreed minimum value, the agreed maximum value, the number of UUIDs in the previous n pending confirmation sets, and the previous single packet size limit when packaging files in bulk, designing a dynamic packetization strategy suitable for the small file one-way blind transmission scenario in this embodiment.

[0042] For example, the client can use a dynamic packaging strategy to generate several multi-part data packets based on each file to be uploaded in the file list. Each multi-part data packet contains at least one multi-part sub-block (in this embodiment, the number of multi-part sub-blocks contained in each multi-part data packet is generally no less than 3). In addition, when designing a dynamic packetization strategy, it is necessary to consider the upper limit of single packet size and the upper limit of single packet quantity. The upper limit of single packet size refers to the size of a single multi-part data packet not exceeding a certain limit. In this embodiment, this limit is taken as 100M (50M to 1G is acceptable, but if the single packet is too large, the cost of retransmission due to transmission failure is high, while if the single packet is too small, it is easy to lead to excessive protocol overhead). The upper limit of single packet quantity needs to be dynamically adjusted within a certain range. When transmitting photo files, the maximum number of single packets is generally 50, but after exceeding 20, the packet loss rate will gradually increase, which will easily lead to retransmission. If the number of files in a single packet is too small, more packetizations are required, which increases the packetization overhead. Therefore, the upper limit of single packet quantity is set to not exceed 20 and not less than 5 (that is, the agreed minimum value is 5 and the agreed maximum value is 20), while the upper limit of single packet quantity is dynamically adjusted according to the specific situation each time packetization is performed.

[0043] Specifically, when using a dynamic packaging strategy, the client can obtain the set maximum size of a single package (e.g., 100MB), then obtain the number of UUIDs in the current set to be confirmed, and determine the maximum number of packages per package based on the number of UUIDs. Of course, for the first packaging, if the number of UUIDs in the current set to be confirmed is zero, the initial maximum number of packages per package (i.e., the agreed-upon base value, e.g., 10) is directly used as the maximum number of packages per package for this time. For subsequent packaging, the client first checks if the number of UUIDs in the current set to be confirmed satisfies the following conditions: If the number of UUIDs in the current set to be confirmed (e.g., 234) is higher than the first threshold (e.g., 200), and if the previous maximum single packet quantity value (e.g., 10) is greater than the agreed minimum value (e.g., 5), the client reduces the set quantity based on the previous maximum single packet quantity value (in this embodiment, the set quantity is 1 when reducing). If the previous maximum single packet quantity value (e.g., 5) is not greater than the agreed minimum value (e.g., 5), the agreed minimum value (e.g., 5) is determined as the maximum single packet quantity for this time.

[0044] If the number of UUIDs in the current set to be confirmed (e.g., 54) is lower than the second threshold (e.g., 100), the number of UUIDs in the previous n times is obtained (this embodiment uses the previous UUID number as an example; other embodiments may use more times, but preferably no more than 3; for cases where the number of times obtained is insufficient, zero is used to represent the number of UUIDs in the corresponding number of times). If the number of UUIDs in the previous n times is lower than the second threshold (e.g., 100), and the previous single packet quantity limit value (e.g., 10) is less than the agreed maximum value (e.g., 20), a set number is added to the previous single packet quantity limit value (in this embodiment, the set number is also 1 when incrementing). If the number of UUIDs in the previous n times is lower than the second threshold (e.g., 100), and the previous single packet quantity limit value (e.g., 20) is not less than the agreed maximum value (e.g., 20), the agreed maximum value (e.g., 20) is determined as the single packet quantity limit for this time. If the number of UUIDs in the previous n times is not all lower than the second threshold, the previous single packet quantity limit value is determined as the single packet quantity limit for this time.

[0045] When the number of UUIDs in the current set to be confirmed (e.g., 123) is not less than the second threshold (e.g., 100) and not more than the first threshold (e.g., 200), the agreed basic value (e.g., 10) is determined as the upper limit of the single packet quantity for this time. Among them, the agreed minimum value (e.g., 5) < the agreed basic value (e.g., 10) < the agreed maximum value (e.g., 20), and the set quantity (e.g., 1) does not exceed the difference between the agreed minimum value and the agreed basic value (e.g., 10-5=5) and the difference between the agreed basic value and the agreed maximum value (e.g., 20-10=10).

[0046] After determining the maximum number of single packets in this batch, the client can select the files to be packed from the file list in order based on the maximum size and number of single packets (files that have already been packed will be marked as packed, and unmarked files will be packed in order each time a batch is packed), and then pack the files to obtain the multi-part data packet for this batch.

[0047] The packaged multi-part data packets can be placed in a queue. When a multi-part data packet exists in the queue, the client can retrieve it and write it into the socket buffer of the uplink HTTP connection. After the sending action is completed, all UUIDs contained in the multi-part data packet are added to the pending acknowledgment set, and the next loop (retrieving the next multi-part data packet from the queue) is immediately started, sending the next multi-part data packet. Of course, for multiple uplink HTTP connections, multiple lines can send concurrently, but the packet size will be adjusted appropriately. Multi-part data packets sent through each uplink HTTP connection are retrieved from the same queue, and all UUIDs contained in the multi-part data packets sent through each uplink HTTP connection are added to the same pending acknowledgment set.

[0048] This dynamic packaging strategy determines the appropriate upper limit for the current number of single packets by comprehensively considering factors such as the number of UUIDs in the current set of packets awaiting confirmation (which can reflect the server's operating pressure to some extent), the number of UUIDs in the previous set of packets awaiting confirmation, the agreed minimum value, the agreed maximum value, the number of UUIDs in the previous n sets of packets awaiting confirmation, and the previous single packet quantity limit. This dynamic adjustment strategy for the number of packets can take into account the server's operating pressure to a certain extent, determine the appropriate number of packets, and avoid blindly increasing the number of single packets, which would lead to an increase in transmission failure rate and a surge in server pressure. Furthermore, the single packet quantity adjustment mechanism is relatively smooth, effectively avoiding network instability.

[0049] Once the server receives a multi-part data packet from any client via an uplink HTTP connection, it enters the file receiving phase. During this phase, the server primarily receives files uploaded by the client. After writing the data, it sends the file processing result back to the client via a downlink feedback link. This result includes a unique file ID and the file receiving status, which can be either successful or failed.

[0050] Step S21 can be run.

[0051] Step S21: Receive files uploaded by the client, wherein the client blindly sends each file in the file list to the server via an uplink HTTP link.

[0052] In this embodiment, the server can listen to the uplink HTTP connection port, read the data stream, parse the multi-part data packet according to the multi-part specification, determine the UUID, File_Size, and File_Name corresponding to each multi-part sub-block in the multi-part data packet, and write the data to a file, thereby completing the reception of a single multi-part data packet (which may be a single file or multiple files).

[0053] After the writing is complete, the server runs step S22.

[0054] Step S22: After writing is complete, the file processing result is fed back to the client via the downlink feedback link. The file processing result includes the file's unique number and the file receiving status, which is either successful or failed.

[0055] For example, after writing is complete, the server can generate a file processing result {UUID, Success / Fail} and forward it to the internal message queue (MQ), completing the parsing of this multi-part data packet. Then, the server continues parsing the next multi-part data packet, where UUID is the file's unique identifier, and Success / Fail indicates the file reception status: Success means the file upload was successful, and Fail means the file upload failed. Furthermore, after receiving the file processing result from the internal message queue (MQ), the server can provide feedback to the client via a downlink feedback link according to the SSE protocol. The server can provide feedback to the client via the downlink feedback link in an immediate manner (e.g., one result per request) or in a phased manner (e.g., once per second).

[0056] At this point, the client can run step S14.

[0057] Step S14: Receive the file processing result from the server, and based on the file processing result, delete the unique file number with a successful file reception status from the pending confirmation set. The file processing result includes the unique file number and the file reception status, which is either successful or failed.

[0058] In this embodiment, the client can listen to the downlink feedback link port, receive the file processing result from the server, determine the unique file number and file reception status from the file processing result, and then determine the unique file number with a successful file reception status, thereby deleting the unique file number with a successful file reception status from the set to be confirmed.

[0059] After completing a single iteration of the file list at set intervals (e.g., 30 seconds), the client can obtain the UUIDs in the current pending confirmation set (which typically belong to files that failed to upload). At this point, the client identifies the retransmission file corresponding to the UUID in the current pending confirmation set from the file list, clears the pending confirmation set, and then blindly sends each retransmission file in the file list to the server via an uplink HTTP connection. The client also adds the unique file number of the sent retransmission file to the pending confirmation set. This process is repeated until all files in the file list are successfully uploaded.

[0060] Screenshot of the client for batch uploading small files as shown below Figure 4 As shown.

[0061] In summary, this application provides a file upload system and method based on HTTP protocol with separate uplink and downlink connections. This solution obtains a file list (containing multiple files to be uploaded, each with a unique file ID) through the client, establishes an uplink HTTP link and a downlink feedback link, and blindly sends each file in the file list to the server unidirectionally, adding the unique file IDs of the sent files to the pending confirmation set. The server receives the uploaded files from the client, and after writing, sends the file processing result (including the file ID and file reception status) back to the client via the downlink feedback link. The client then receives the file processing result from the server and, based on the result, removes the file IDs with a successful reception status from the pending confirmation set. This solution breaks the native HTTP protocol's strong "request-response" synchronous mode, separating the upload of binary file data and the feedback of the upload result status into two independent communication channels (which can be two TCP connections, or two independent logical streams on one connection). This transforms the uplink channel into a "one-way blind transmission" mode, completely eliminating the RTT (round-trip time) window caused by waiting for ACK packets. Meanwhile, this solution changes the client's sending logic from "send-block-receive-judgment-resend" to "loop read-continuous send-update local status table." The client's sending thread does not perform blocking read operations at all, relying solely on the underlying TCP protocol, thus maximizing bandwidth utilization during the sending phase. Furthermore, by using a unique file number in conjunction with a set of pending confirmations, the server's file reception status can be verified, ensuring the reliability and completeness of file uploads (verifying which files the server has already received, avoiding upload omissions due to various transmission failures).

[0062] During file uploads on the client side, a single-file packaging approach can be used. However, to reduce connection overhead and improve throughput, a dynamic packaging strategy is designed. In one-way blind transmission, the actual server load is ignored, and sending large batches of files without considering server capacity can lead to a significant increase in transmission failure rate. Therefore, this solution, when packaging files in batches, determines the appropriate upper limit for the current single-file quantity by comprehensively considering the number of UUIDs in the current pending confirmation set (which reflects server load to some extent), the number of UUIDs in the previous pending confirmation set, the agreed minimum value, the agreed maximum value, the number of UUIDs in the previous n pending confirmation sets, and the previous single-packet quantity limit. This dynamic adjustment strategy considers server load, determines an appropriate number of files to package, and avoids the increased transmission failure rate and surge in server load caused by blindly increasing the number of files per package. Furthermore, the single-packet quantity adjustment mechanism is relatively smooth, effectively preventing network instability.

[0063] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A file upload system based on HTTP protocol with separate uplink and downlink connections, characterized in that, include: The client is used to obtain a file list, establish an uplink HTTP link and a downlink feedback link, send each file in the file list to the server in a one-way blind manner, and include the unique file number of the sent file in the set to be confirmed. The file list contains multiple files to be uploaded, each file corresponding to a unique file number. One-way blind sending means that the sending end does not wait for the receiving end to respond after completing the previous file sending task. The server is used to receive files uploaded by clients. After writing, it sends the file processing result back to the client via a downlink feedback link. The file processing result includes a unique file number and the file receiving status, which is either successful or failed. The client is also used to receive the file processing results from the server and, based on the file processing results, remove the unique file number with a successful file reception status from the pending confirmation set.

2. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 1, characterized in that, The client is specifically used for: Iterate through the file list, generate a unique file ID (UUID) for each file to be uploaded, and construct a multi-part data packet. The multi-part data packet contains several multi-part sub-blocks, each corresponding to a file to be uploaded. Each multi-part sub-block contains a header and a body. The header includes the file's UUID, File_Size, and File_Name. The UUID is the unique file ID, the File_Size is the file size, and the File_Name is the file name. Retrieve the multi-part data packet from the queue, write it into the socket buffer of the upstream HTTP connection, and after the sending action is completed, add all the UUIDs contained in the multi-part data packet to the set to be confirmed, and immediately enter the next loop to send the next multi-part data packet.

3. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 2, characterized in that, The client is specifically used for: Generate a multi-part data packet for each file to be uploaded in the file list, and each multi-part data packet contains a multi-part sub-block; Alternatively, based on each file to be uploaded in the file list, a dynamic packaging strategy can be used to generate several multi-part data packets, each containing at least one multi-part sub-block.

4. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 3, characterized in that, When using a dynamic packaging strategy, the client is specifically used for: Get the set maximum size of a single packet; Get the number of UUIDs in the current set to be confirmed, and determine the upper limit of the number of packets per package based on the number of UUIDs; Based on the maximum size and maximum number of packages per file, the files to be packaged are determined sequentially from the file list. The files in this package are then packaged to obtain the multi-part data package.

5. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 4, characterized in that, The client is specifically used for: If the number of UUIDs in the current set to be confirmed is higher than the first threshold, if the previous maximum number of packets per transaction was greater than the agreed minimum number, the number of packets per transaction will be reduced by the set number based on the previous maximum number of packets per transaction. If the previous maximum number of packets per transaction was not greater than the agreed minimum number, the agreed minimum number will be determined as the maximum number of packets per transaction for this transaction. If the number of UUIDs in the current set to be confirmed is lower than the second threshold, obtain the number of UUIDs from the previous n times. If the number of UUIDs from the previous n times is lower than the second threshold, and the previous single packet quantity limit is less than the agreed maximum value, increase the set quantity based on the previous single packet quantity limit. If the number of UUIDs from the previous n times is lower than the second threshold, and the previous single packet quantity limit is not less than the agreed maximum value, determine the agreed maximum value as the single packet quantity limit for this time. If the number of UUIDs from the previous n times is not all lower than the second threshold, determine the previous single packet quantity limit as the single packet quantity limit for this time. When the number of UUIDs in the current set to be confirmed is not less than the second threshold and not more than the first threshold, the agreed basic value is determined as the upper limit of the number of packets in this case. The agreed minimum value < agreed basic value < agreed maximum value, and the set number does not exceed the difference between the agreed minimum value and the agreed basic value and the difference between the agreed basic value and the agreed maximum value.

6. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 2, characterized in that, The server is specifically used for: Listen to the upstream HTTP connection port and read the data stream; The multi-part data packet is parsed according to the multi-part specification. The UUID, File_Size, and File_Name corresponding to each multi-part sub-block in the multi-part data packet are determined and the data is written to a file. After writing is completed, the file processing result {UUID, Success / Fail} is forwarded to the internal message queue MQ to complete the parsing of this multi-part data packet and continue to the parsing of the next multi-part data packet. Here, UUID is the unique number of the file, and Success / Fail is the file receiving status. Success indicates that the file upload was successful and Fail indicates that the file upload failed. Receive file processing results from the internal message queue (MQ) and send them back to the client via a downlink feedback link according to the SSE protocol.

7. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 6, characterized in that, The client is specifically used for: Listen to the downlink feedback connection port and receive the file processing results from the server. From the file processing results, determine the unique file number and file reception status, and determine the unique file number of the file with a successful reception status; Remove the unique file IDs of files whose file reception status is successful from the pending confirmation set.

8. The file upload system based on HTTP protocol with separate uplink and downlink connections according to claim 1, characterized in that, The client is also used for: After completing the traversal of the file list at set intervals, retrieve the UUIDs from the current set of files awaiting confirmation. Identify the retransmission file corresponding to the UUID in the current set to be confirmed from the file list, and clear the set to be confirmed. Each retransmitted file in the file list is sent blindly to the server via an uplink HTTP link, and the unique file number of each retransmitted file is included in the set to be confirmed.

9. A file upload method based on HTTP protocol with separate uplink and downlink connections, characterized in that, A client applied to a file upload system based on the HTTP protocol with separate uplink and downlink connections, as described in any one of claims 1-8, the method comprising: Get the file list, which contains multiple files to be uploaded, each file having a unique file number; Establish upstream HTTP links and downstream feedback links; Each file in the file list is sent to the server in a one-way blind manner, and the unique file number of the sent file is included in the set to be confirmed. One-way blind sending means that the sender does not wait for the receiver's response after completing the previous file sending task. The system receives the file processing result from the server and, based on the result, removes the unique file IDs with a successful file reception status from the pending confirmation set. The file processing result includes the unique file ID and the file reception status, which is either successful or failed.

10. A file upload method based on HTTP protocol with separate uplink and downlink connections, characterized in that, A server applied to a file upload system based on the HTTP protocol with separate uplink and downlink connections, as described in any one of claims 1-8, wherein the method comprises: Receive files uploaded by the client, wherein the client blindly sends each file in the file list to the server via an uplink HTTP link; After the writing is complete, the file processing result will be sent back to the client via a downlink feedback link. The file processing result includes the file's unique number and the file reception status, which is either successful or failed.

Images