File fragment transmission method and device, server, medium and program product
By using an SFTP session channel to transmit file fragments and verify digest values in real time during file upload, the disk I/O and network overhead problems of large file transfer in existing technologies are solved, achieving efficient file transfer and storage.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA CONSTRUCTION BANK
- Filing Date
- 2023-05-22
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies suffer from high disk I/O costs, significant time overhead, network overhead, and high costs associated with managing junk files during file uploads, especially when enterprise-level servers are transferring large files from local disks to remote storage services.
By establishing a secure file transfer protocol (SFTP) session channel, file fragments are transferred to the target storage server in real time. During the transfer process, the consistency of the message digest values of the file fragments is verified until all fragments are combined into the target file, at which point the session channel is closed.
It reduces local disk I/O and network overhead on the receiving server, while also reducing junk file management costs and improving file transfer efficiency.
Smart Images
Figure CN116389515B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a file fragmentation transmission method, apparatus, server, medium and program product. Background Technology
[0002] The Hypertext Transfer Protocol (HTTP) is a request-response protocol commonly used for interaction between browsers and servers. When using a browser, users typically utilize HTTP technology to download files from or upload files to a server.
[0003] During file uploads, after receiving a file, especially for enterprise-level servers, large files typically need to be transferred to a remote dedicated storage service. This transfer process involves first storing the original file on the local disk, and then using the data center's internal network to send the entire file from the local disk to the remote dedicated storage service for permanent storage. This entire transfer process incurs significant disk I / O costs, additional time and network overhead, and the cost of managing junk files on the receiving end. Summary of the Invention
[0004] To address the aforementioned issues, namely the high time overhead and high garbage management costs incurred during the file transfer and dumping process, this application provides a file fragmentation transmission method, apparatus, server, medium, and program product.
[0005] According to a first aspect of this application, a method for transmitting file fragments is provided, comprising:
[0006] In response to a file fragment transfer request for the target file, a Secure File Transfer Protocol (SFTP) session channel is established with the target storage server;
[0007] For each file fragment corresponding to the target file received, verify whether the information digest value of the file fragment is consistent with the preset information digest value;
[0008] When the information digest value of the file fragment matches the preset information digest value, the file fragment is transmitted to the target storage server based on the SFTP session channel.
[0009] In response to the completion of transmission of all file fragments of the target file, when the total first message digest value of all file fragments matches the total second message digest value of the target file, the file fragments in the target storage server are combined to obtain the target file, and the SFTP session channel is closed.
[0010] In one implementation, before transferring the file fragments to the target storage server based on the SFTP session channel, the method further includes:
[0011] Determine whether the storage file corresponding to the target file has already been created on the target storage server;
[0012] The process of transmitting the file fragments to the target storage server based on the SFTP session channel includes:
[0013] When a storage file corresponding to the target file has been created in the target storage server, obtain the storage file name and storage path corresponding to the storage file.
[0014] Based on the SFTP session channel, the file is fragmented and written to the corresponding storage file on the target storage server according to the storage file name and storage path.
[0015] In one implementation, after determining whether a storage file corresponding to the target file has already been created in the target storage server, the method further includes:
[0016] If no storage file corresponding to the target file is created in the target storage server, the file information of the target file is read, the file information includes basic file information and / or file data information, and the storage space required for the target file is determined based on the file information;
[0017] A first instruction message is sent to the target storage server. The first instruction message carries the storage space information required by the target file, so that the target storage server can create a storage file based on the first instruction message and determine the storage file name and storage path corresponding to the storage file.
[0018] In one implementation, before transferring the file fragments to the target storage server based on the SFTP session channel, the method further includes:
[0019] The system locks the current file creation permissions based on a global lock control and synchronizes the corresponding permission lock information with the target storage server, so that the target storage server creates a storage file based on the instruction message when creating a storage file.
[0020] In one embodiment, the method further includes:
[0021] In response to the completion of the creation of the storage file on the target storage server, the current file creation permission is unlocked, and the fragment write permission of the storage file is locked, so that when the file fragment is written to the storage file corresponding to the target storage server, the file fragment is written to the storage file only once.
[0022] In one implementation, the stored file is a file with a preset file size created by the target storage server;
[0023] The step of writing the file fragments to the corresponding storage file on the target storage server based on the SFTP session channel, according to the storage file name and storage path, includes:
[0024] Based on the SFTP session channel, the file is fragmented and written to the absolute offset of the storage file corresponding to the target storage server according to the storage file name and storage path. The absolute offset is determined based on the actual offset of the storage file and the preset file size.
[0025] In one implementation, the file fragment transfer request is an HTTP PUT request, and the method further includes:
[0026] When the information digest value of the file fragment is inconsistent with the preset information digest value, a second indication information is sent to the requesting end that initiated the file fragment transmission request. The second indication information is used to instruct the requesting end to resend the file fragment transmission request for the corresponding file fragment. The resent fragment transmission request carries the fragment update file corresponding to the file fragment.
[0027] According to a second aspect of this application, a file fragmentation transmission apparatus is provided, comprising:
[0028] The channel establishment module is configured to establish an SFTP session channel with the target storage server in response to a file fragment transfer request for the target file.
[0029] The verification module is configured to verify whether the message digest value of each file fragment corresponding to the target file is consistent with a preset message digest value for each received file fragment.
[0030] The transmission module is configured to transmit the file fragment to the target storage server based on the SFTP session channel when the information digest value of the file fragment is consistent with the preset information digest value.
[0031] The shutdown module is configured to, in response to the completion of the transmission of all file fragments of the target file, when the total first message digest value of all file fragments matches the total second message digest value of the target file, combine the file fragments in the target storage server to obtain the target file, and close the SFTP session channel.
[0032] In one embodiment, the device further includes:
[0033] The judgment module is configured to determine whether a storage file corresponding to the target file has been created in the target storage server.
[0034] The transmission module includes:
[0035] The acquisition unit is configured to acquire the storage file name and storage path corresponding to the storage file when a storage file corresponding to the target file has been created in the target storage server.
[0036] The writing unit is configured to write the file into fragments according to the storage file name and storage path to the storage file corresponding to the target storage server based on the SFTP session channel.
[0037] In one embodiment, the device further includes:
[0038] The reading module is configured to read the file information of the target file when no storage file corresponding to the target file is created in the target storage server. The file information includes basic file information and / or file data information, and the module determines the storage space required for the target file based on the file information.
[0039] The first instruction module is configured to send a first instruction message to the target storage server. The first instruction message carries storage space information required by the target file, so that the target storage server creates a storage file based on the first instruction message and determines the storage file name and storage path corresponding to the storage file.
[0040] In one implementation, the stored file is a file with a preset file size created by the target storage server;
[0041] The writing unit is specifically configured to, based on the SFTP session channel, write the file fragments to the absolute offset of the storage file corresponding to the target storage server according to the storage file name and storage path, wherein the absolute offset is determined according to the actual offset of the storage file and the preset file size.
[0042] According to a third aspect of this application, a server is provided, comprising: a memory and a processor;
[0043] The memory stores computer-executed instructions;
[0044] The processor executes computer execution instructions stored in the memory, causing the server to execute the file fragmentation transfer method.
[0045] According to a fourth aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, are used to implement the file fragmentation transmission method described above.
[0046] According to a fifth aspect of this application, a computer program product is provided, the computer program product including computer program code, which, when run on a computer, causes the computer to perform the file fragmentation transmission method.
[0047] Understandably, the file chunking transmission method, apparatus, server, medium, and program products provided in this application establish a Secure File Transfer Protocol (SFTP) session channel with the target storage server in response to a file chunking transmission request for a target file. Each time a file chunk corresponding to the target file is received, the message digest value of the file chunk is verified to be consistent with a preset message digest value. When the message digest value of the file chunk is consistent with the preset message digest value, the file chunk is transmitted to the target storage server based on the SFTP session channel. In response to the completion of the transmission of all file chunks of the target file, when the total first message digest value of all file chunks matches the total second message digest value of the target file, the file chunks in the target storage server are combined to obtain the target file, and the SFTP session channel is closed. In the process of file chunking transmission, this application establishes an SFTP session channel between the receiving server and the target storage server. In the large file chunking upload process, there is no need for local caching; the process of transmitting file chunks to the backend file storage server in real time reduces the local disk I / O overhead, additional time overhead, and network overhead of the receiving server, while also reducing the cost of managing junk files at the receiving end. Attached Figure Description
[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0049] Figure 1 A flowchart illustrating a file fragmentation transmission method provided in an embodiment of this application;
[0050] Figure 2 One of the flowcharts for another file fragmentation transmission method provided in the embodiments of this application;
[0051] Figure 3 A second schematic flowchart illustrating another file fragmentation transmission method provided in this application embodiment;
[0052] Figure 4A flowchart illustrating yet another file fragmentation transmission method provided in this application embodiment;
[0053] Figure 5 A flowchart illustrating a file fragmentation transmission method provided for an exemplary embodiment of this application;
[0054] Figure 6 A schematic diagram of a file fragmentation transmission device provided in an embodiment of this application;
[0055] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application.
[0056] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0057] In one possible application scenario, the system includes a terminal device and a server electrically connected to the terminal device. In this embodiment, a browser on the terminal device is used as the requesting end or client. The terminal device interacts with the server via a network. Users can upload large files to the server using the Hypertext Transfer Protocol (HTTP) technology in their browser. Large files specifically refer to files that occupy a significant amount of storage space, such as common video files or application software installation files.
[0058] The client devices may include, but are not limited to, computers, smartphones, tablets, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, portable computers, in-vehicle computers, wearable devices, desktop computers, set-top boxes, smart TVs, and so on.
[0059] A server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.
[0060] In real-world applications, when users upload large files from their browsers to servers using HTTP technology, they often encounter a series of problems, including slow upload speeds, file size limits imposed by gateways, interruptions during uploads, and incomplete files due to lost data packets during HTTP transmission. Existing technologies address the issue of large files not being able to be uploaded all at once by splitting the file into many smaller, identically sized files and validating these smaller files to obtain the larger file. However, this introduces new problems. After obtaining the smaller files locally and then the larger file, the receiving end (server) typically needs to transfer the larger file to a remote storage server. During this transfer process, the original file is first written to the local disk, and then the entire file is sent from the local disk to a dedicated remote storage service for permanent storage using the data center's internal network. This results in significant disk I / O costs, additional time and network overhead, and a large amount of redundant junk files on the receiving end.
[0061] In view of this, the file chunking transmission method, apparatus, server, medium, and program products provided in this embodiment, combined with Secure File Transfer Protocol (SFTP) technology, establish an SFTP session channel with the target storage server in response to a file chunking transmission request for the target file. Each time a file chunk corresponding to the target file is received, the message digest value of the file chunk is verified to be consistent with a preset message digest value. When the message digest value of the file chunk is consistent with the preset message digest value, the file chunk is transmitted to the target storage server based on the SFTP session channel. In response to the completion of the transmission of all file chunks of the target file, when the total first message digest value of all file chunks matches the total second message digest value of the target file, the file chunks in the target storage server are combined to obtain the target file, and the SFTP session channel is closed. During this process, the establishment of an SFTP session channel between the receiving server and the target storage server during file chunking transmission enables real-time transmission of chunks to the backend file storage server after they are uploaded locally, thereby reducing the local disk I / O overhead of the receiving server and reducing the overall network overhead of the system.
[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar components or components having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.
[0063] It should be noted that the file fragmentation transmission method of this application can be used in the fintech field. It can also be used in any field other than fintech; the application field of the file fragmentation transmission method of this application is not limited.
[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.
[0065] The application scenarios of this application have been briefly described above. The following describes the file fragmentation transmission method provided by the embodiments of this application in detail, taking the application to a server, specifically the application to a receiving server.
[0066] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a file fragmentation transmission method provided in an embodiment of this application, including steps S101-S104.
[0067] Step S101: In response to the file fragmentation transfer request of the target file, establish a Secure File Transfer Protocol (SFTP) session channel with the target storage server.
[0068] In this embodiment, the target file is the entire file that the client needs to transmit, i.e., a large file, which includes at least two or more file fragments. In one implementation, the file fragment transmission request can use a request method other than HTTP PUT, which can effectively reduce the generation of temporary files during fragment transmission.
[0069] In this embodiment, the receiving server can utilize the open-source JSCH component to create an SFTP operation API interface and a corresponding SFTP protocol client, and establish an SFTP channel with the target storage server to interact with the target storage server using this SFTP channel. It is understood that the JSCH component is an SFTP protocol client component; in some embodiments, other SFTP protocol client components may also be used besides the aforementioned JSCH component.
[0070] Understandably, in response to, is used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.
[0071] In one example, before receiving a file fragment transfer request, the receiving server first reaches a consensus with the client to perform fragment transfer. Specifically, the client first initiates a file transfer request to the server using the HTTP POST request method. The receiving server obtains and records the basic information of the target file, such as the full-text MD5 value of the target file and the unique identifier TaskID, and sends instruction information to the client. The client further generates fragments based on the instruction information and initiates a fragment transfer request to the receiving server.
[0072] Step S102: For each file fragment corresponding to a target file received, verify whether the information digest value of the file fragment is consistent with the preset information digest value.
[0073] Specifically, each time the server receives a file fragment corresponding to a target file, it reads the information digest value of that file fragment and verifies the file fragment. If the verification fails, it can instruct the client to retransmit the corresponding file fragment.
[0074] Optionally, when generating file fragments, the terminal device calculates an initial message digest value (i.e., a preset message digest value) for each file fragment. When the server receives a file fragment, it uses this initial message digest value to verify the received file fragment. By verifying whether the message digest values of the two fragments are the same, the integrity of the fragment transmission is determined.
[0075] Optionally, the client can use a round-robin method to transmit file fragments. That is, the client identifies whether there are any unsent file fragments among all the generated file fragments. If so, the client sends a file fragment transmission request corresponding to that file fragment to the server until all file fragments have been sent.
[0076] Step S103: When the information digest value of the file fragment is consistent with the preset information digest value, the file fragment is transmitted to the target storage server based on the SFTP session channel.
[0077] In this embodiment, after verifying the file fragments, instead of combining the individual file fragments into a complete file and then transferring the complete file to the storage server, as is done in the prior art, the file fragments that have passed verification are directly transferred to the target storage server during the file fragment transmission process. This process eliminates the need to store or cache file fragments on the receiving server, effectively reducing disk I / O costs on the receiving server side, while also avoiding additional time and network overhead, and simultaneously reducing the cost of managing junk files.
[0078] Optionally, the client can use a round-robin method to transmit file fragments. That is, the client identifies whether there are any unsent file fragments among all the generated file fragments. If so, it sends a file fragment transmission request corresponding to that file fragment to the receiving server and directly dumps it to the target storage server until all file fragments have been sent.
[0079] Step S104: In response to the completion of the transmission of all file fragments of the target file, when the total value of the first message digest of all file fragments matches the total value of the second message digest of the target file, the file fragments in the target storage server are combined to obtain the target file, and the SFTP session channel is closed.
[0080] In this embodiment, the total second message digest value of the target file can be calculated before the fragmented transmission. By verifying the total MD5 value of the file fragments and the total MD5 value of the target transmitted file, it is determined whether all file fragments have been transmitted. When all file fragments have been transmitted, they are combined to form a large file, and the SFTP channel is closed to complete the file transmission and dumping process.
[0081] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating another file fragmentation transmission method provided in this application embodiment. Based on the above embodiment, this embodiment creates a storage file in the target storage server and writes the file fragments into the corresponding storage file to achieve file fragmentation writing. Specifically, before transmitting the file fragments to the target storage server based on the SFTP session channel in step S103, the method may further include step S201, and step S103 is further divided into step S103a and step S103b.
[0082] Step S201: Determine whether the storage file corresponding to the target file has been created in the target storage server. If the storage file corresponding to the target file has been created in the target storage server, then execute step S103a; otherwise, end the process or create the storage file. The method of creating the storage file is described in detail later and will not be repeated here.
[0083] Step S103a: Obtain the storage file name and storage path corresponding to the storage file;
[0084] Step S103b: When the information digest value of the file fragment is consistent with the preset information digest value, the file fragment is written to the corresponding storage file of the target storage server based on the SFTP session channel, according to the storage file name and storage path.
[0085] As is understandable, a storage file is a file used to write the file segment corresponding to the target file.
[0086] In one implementation, the existence of the storage file is determined by calling the channel.stat method. If the method returns an exception with the message "No such file", the storage file is determined to be non-existent; otherwise, the storage file exists.
[0087] In one implementation, such as Figure 3 As shown, after determining in step S201 whether the storage file corresponding to the target file has been created in the target storage server, the steps 301 and S302 may also be included.
[0088] Step S301: If no storage file corresponding to the target file is created in the target storage server, read the file information of the target file. The file information includes basic file information and / or file data information, and determine the storage space required for the target file based on the file information.
[0089] Step S302: Send a first instruction message to the target storage server. The first instruction message carries the storage space information required by the target file, so that the target storage server can create a storage file based on the first instruction message and determine the storage file name and storage path corresponding to the storage file.
[0090] Specifically, this embodiment reads the file information of the target file, including basic file information (including MD5 digest information) and file data information, and then determines the storage space required for the target file based on the file information. By creating a storage file of the appropriate size in the storage server, the storage file name and path of the storage server are further determined.
[0091] In this embodiment, step S302 determines the storage file name and storage path, and then executes step S103b to store the fragmented file into the corresponding storage file according to the storage file name and storage path.
[0092] In one implementation, by locking file creation permissions when creating a storage file to prevent duplicate file creation, the method may further include the following before step S103, which involves transferring file fragments to the target storage server based on the SFTP session channel:
[0093] The system locks the current file creation permissions using a global lock control and synchronizes the corresponding permission locking information with the target storage server, so that the target storage server creates a storage file based on the instruction message when creating a storage file.
[0094] In one implementation, the global lock control can be the redission-rlock control of the Redis distributed lock. In other implementations, other global lock spaces can also be used. This embodiment does not make specific limitations on this.
[0095] In this embodiment, when the storage server creates a file, a global lock control is used to lock the file creation permission. The storage server can only create one storage file, which can effectively prevent the duplicate creation of storage files, so as to facilitate the effective writing of file fragments.
[0096] Furthermore, after file creation is complete, locking write permissions for each fragment effectively prevents duplicate writes to the fragments. Specifically, the method may also include the following steps:
[0097] In response to the completion of the creation of the storage file on the target storage server, the current file creation permission is unlocked, and the fragment write permission of the storage file is locked, so that when the file fragment is written to the corresponding storage file on the target storage server, the file fragment is written to the storage file only once.
[0098] In this embodiment, after the storage file is created, the file creation permission is unlocked, while the fragment write permission of the storage file is locked. When performing fragment write, only one file fragment can be written to avoid repeated writing of a file fragment. The fragment write permission can also be locked using the redission-rlock control or other controls.
[0099] In one implementation, considering that SFTP does not support random access to files, but only append access, and each fragment needs to be written to a specified offset in the storage file through random access, this embodiment utilizes append mode, setting the absolute offset to a negative offset form of offset - size, to indirectly achieve random access to files. Specifically, the storage file is a file with a preset file size created by the target storage server;
[0100] In step S103b, based on the SFTP session channel, file fragments are written to the corresponding storage files on the target storage server according to the storage file name and storage path. Specifically, the steps are as follows:
[0101] Based on the SFTP session channel, the file is fragmented and written to the absolute offset of the corresponding storage file on the target storage server according to the storage file name and storage path. The absolute offset is determined based on the actual offset of the storage file and the preset file size.
[0102] Understandably, the offset in append-only access can be negative. This embodiment utilizes append mode, setting the absolute offset to a negative offset of offset - size, to indirectly achieve random access to the file. Specifically, a storage file of size size is created (where the file size is greater than 1), and the file stream is opened in append mode with the opening offset set to size - 1. An arbitrary byte is written at this offset to determine the file size, and then the file is closed. During fragment writing, the created storage file is opened in append mode at the absolute offset (actual offset - size), and the current file fragment is appended.
[0103] In this embodiment, the previously created file is opened by appending. The offset of the opened file is (actual offset - size) of the current fragment, forming a negative value relative to the end of the file. From this offset, the uploaded fragment data read from the file fragment transfer request is written to the storage file in 1024-byte units, and then written to the storage file until the fragment writing is completed. This process uses append access mode to indirectly achieve random access for fragment writing.
[0104] Please refer to Figure 4 , Figure 4This is a flowchart illustrating another file fragmentation transmission method provided in this application embodiment. Based on the above embodiment, the file fragmentation transmission request in this embodiment is an HTTP PUT request, and the corresponding MD5 file fragments are updated by sending fragmentation update files. This can effectively reduce the generation of temporary files during fragmentation transmission. Specifically, in addition to the above steps S101-S104, the method of this embodiment may also include the following step S401.
[0105] Step S401: When the information digest value of a file fragment is inconsistent with the preset information digest value, a second indication information is sent to the requesting end that initiated the file fragment transmission request. The second indication information is used to instruct the requesting end to resend the file fragment transmission request for the corresponding file fragment. The resent fragment transmission request carries the fragment update file corresponding to the file fragment.
[0106] In this embodiment, each time the server receives a file fragment, it reads the information digest value of that file fragment and verifies the file fragment. If the verification fails, it instructs the client to retransmit the corresponding file fragment, thereby updating the file fragment. Furthermore, using the HTTP PUT method can effectively avoid the accumulation of temporary files and alleviate system storage pressure.
[0107] To facilitate understanding of the embodiments of this application, the following description is provided in conjunction with an exemplary embodiment of this application, such as... Figure 5 As shown, the process includes the following:
[0108] Step S501: The user selects the target file on the client.
[0109] Step S502: If the user does not select a target file, the client can use a JS script to obtain the target file selected by the user through the onchange event.
[0110] Step S503: The client reads the target file size, length, etc.
[0111] Step S504: The client calculates the MD5 hash of the target file. Specifically, it can be calculated using spark-md5.js.
[0112] Understandably, the steps S501-S504 above are the stages for the client to read and verify the target file.
[0113] Step S505: The client creates a unique string TaskID for the target file.
[0114] Step S506: The client sends the upload / begin request (file transfer request, carrying basic information about the target file) from step 1 to the receiving server in the form of an HTTP POST request.
[0115] Specifically, the client initiates an HTTP POST request, submitting information such as the filename, the full-text MD5 value of the file, the file length, and the TaskID, a unique identifier for the process, to the receiving server.
[0116] Step S507: Receive the basic information of the target file recorded by the server and store it in the target database.
[0117] Step S508: The client performs a fragmentation operation on the target file and initializes the fragments, for example, the total number of fragments is a, and the initial fragment number is set to 0, so as to facilitate recording the fragmentation transmission process.
[0118] Step S509: Client sets execution conditions: Determine if there are any unsent fragments. If there are, proceed to step S510; otherwise, proceed to step S520.
[0119] Step S510: The client calculates the file fragment MD5 and file fragment size, etc.
[0120] Step S511: The client sends the upload / slice request (file fragment transfer request) from step 2 to the receiving server in the form of an HttpPut request.
[0121] Specifically, the client can use HTTP PUT requests to repeatedly upload fragmented data to the receiving server after the fragments have been divided.
[0122] Step S512: The receiving server performs MD5 verification on the file fragments.
[0123] Step S513: After the MD5 value verification of the file fragment passes, the receiving server locks the current (storage) file creation (operation) permissions through a global lock control (e.g., redission-rlock).
[0124] Step S514: The storage server confirms the file creation status.
[0125] Step S515: The receiving server determines whether the current file has been created based on the file creation status of the storage server. If the file has not been created, the receiving server is instructed to create the file. If the file has been created, step S517 is executed.
[0126] Specifically, if the file does not exist, the SFTP write method with a specified offset can be called to create a file of a specified size, and the storage file size can be determined based on the size of the passed-in file.
[0127] Step S516: The storage server creates a file according to the instructions of the receiving server.
[0128] Step S517: Unlock the creation operation permission for the current file and lock the write (write operation) permission for the current file fragment.
[0129] Locking file creation permissions means locking the global file creation permission for the source file's MD5 digest value; unlocking file creation operation permissions means unlocking the global file creation permission for the source file's MD5 digest value.
[0130] Step S518: With write permissions for file fragments, the storage server writes file fragments into the storage file.
[0131] Step S519: Unlock write permissions for the current file segment to facilitate writing to subsequent file segments.
[0132] Understandably, steps S505-S519 above are the fragmentation transmission and dumping stages.
[0133] Step S520: The client sends a request to Step 3: upload / finish request (file fragment transfer completion request);
[0134] Step S521: The storage server calculates the total MD5 hash of the file fragments and returns the result of successful or failed transmission of the target file to the client.
[0135] Specifically, one approach to calculating the MD5 digest of files on the storage server side involves remotely calling the MD5sum command, receiving the return value, and parsing it. On the receiving server, the JSch component can also remotely call the storage server's md5sum command to obtain the total MD5 digest value of each file fragment.
[0136] Step S522: Process terminated.
[0137] Understandably, steps S520-S522 above represent the end stage of the process.
[0138] This application also provides a file fragmentation transmission device, such as... Figure 6 As shown, it includes:
[0139] Channel establishment module 61 is configured to establish an SFTP session channel with the target storage server in response to a file fragment transfer request of the target file;
[0140] The verification module 62 is configured to verify whether the message digest value of each received file fragment of a target file is consistent with the preset message digest value.
[0141] The transmission module 63 is configured to transmit the file fragment to the target storage server based on the SFTP session channel when the information digest value of the file fragment is consistent with the preset information digest value.
[0142] The shutdown module 64 is configured to, in response to the completion of the transmission of all file fragments of the target file, combine the file fragments in the target storage server to obtain the target file when the total first message digest value of all file fragments matches the total second message digest value of the target file, and close the SFTP session channel.
[0143] In one embodiment, the device further includes:
[0144] The judgment module is configured to determine whether the storage file corresponding to the target file has already been created on the target storage server.
[0145] Transmission module 63 includes:
[0146] The acquisition unit is configured to acquire the storage file name and storage path corresponding to the storage file when the storage file corresponding to the target file has been created in the target storage server.
[0147] The write unit is configured to write file fragments to the corresponding storage files on the target storage server according to the storage file name and storage path, based on the SFTP session channel.
[0148] In one embodiment, the device further includes:
[0149] The reading module is configured to read the file information of the target file when no storage file corresponding to the target file has been created on the target storage server. The file information includes basic file information and / or file data information, and the required storage space for the target file is determined based on the file information.
[0150] The first instruction module is configured to send a first instruction message to the target storage server. The first instruction message carries the storage space information required by the target file, so that the target storage server can create a storage file based on the first instruction message and determine the storage file name and storage path corresponding to the storage file.
[0151] In one embodiment, the device further includes:
[0152] The first permission locking module is configured to lock the current file creation permission based on a global lock control and synchronize the corresponding permission locking information with the target storage server, so that the target storage server creates a storage file based on the instruction message when creating a storage file.
[0153] In one embodiment, the device further includes:
[0154] The second permission locking module is configured to unlock the current file creation permission and lock the fragment write permission of the storage file in response to the completion of the storage file creation on the target storage server, so that when the file fragment is written to the corresponding storage file on the target storage server, the file fragment is written to the storage file only once.
[0155] In one implementation, the storage file is a file with a preset file size created by the target storage server;
[0156] The writing unit is specifically configured to write file fragments to the absolute offset of the corresponding storage file on the target storage server based on the storage file name and storage path, using the SFTP session channel. The absolute offset is determined based on the actual offset of the storage file and the preset file size.
[0157] In one implementation, the file fragment transfer request is an HTTP PUT request, and the apparatus further includes:
[0158] The second instruction module is configured to send a second instruction message to the requesting end that initiated the file fragment transmission request when the information digest value of the file fragment is inconsistent with the preset information digest value. The second instruction message is used to instruct the requesting end to resend the file fragment transmission request for the corresponding file fragment. The resent fragment transmission request carries the fragment update file corresponding to the file fragment.
[0159] It should be noted that the apparatus provided in this application can implement all the method steps implemented by the server in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.
[0160] This application also provides a server, such as... Figure 7 As shown, it includes: a memory 71 and a processor 72;
[0161] Memory 71 stores instructions executed by the computer;
[0162] The processor 72 executes computer execution instructions stored in the memory 71, causing the server to execute a file fragmentation transfer method.
[0163] It should be noted that the server provided in this application can implement all the method steps in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.
[0164] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the file fragmentation transmission method described above.
[0165] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to execute a database update method.
[0166] It should be noted that the computer media and program products provided in this application can implement all the method steps implemented in the above method embodiments and achieve the same technical effect. Therefore, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail here.
[0167] This application also provides a chip, including a memory and a processor. The memory is used to store computer programs, and the processor is used to call and run the computer programs from the memory to execute a file fragmentation transfer method.
[0168] It should be noted that the chip provided in this application can implement all the method steps in the above method embodiments and achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiments will not be described in detail here.
[0169] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).
[0170] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.
[0171] Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0172] In the description of the embodiments of this application, the term "and / or" merely indicates a relationship describing the associated objects, meaning that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the term "at least one" indicates any combination of at least two of a plurality of options, for example, including at least one of A, B, and C, which can represent any one or more elements selected from a set including communication between A, B, and C. Furthermore, the term "multiple" means two or more, unless otherwise precisely specified.
[0173] In the description of the embodiments of this application, the terms "first," "second," "third," "fourth," etc. (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for transmitting file fragments, characterized in that, include: In response to a file fragment transfer request for the target file, a Secure File Transfer Protocol (SFTP) session channel is established with the target storage server; For each file fragment corresponding to the target file received, verify whether the information digest value of the file fragment is consistent with the preset information digest value; When the information digest value of the file fragment matches the preset information digest value, the file fragment is transmitted to the target storage server based on the SFTP session channel. In response to the completion of transmission of all file fragments of the target file, when the total first message digest value of all file fragments matches the total second message digest value of the target file, the file fragments in the target storage server are combined to obtain the target file, and the SFTP session channel is closed; Before transferring the file fragments to the target storage server based on the SFTP session channel, the process also includes: Determine whether the storage file corresponding to the target file has already been created on the target storage server; The process of transmitting the file fragments to the target storage server based on the SFTP session channel includes: When a storage file corresponding to the target file has been created in the target storage server, obtain the storage file name and storage path corresponding to the storage file. Based on the SFTP session channel, the file is fragmented and written to the absolute offset of the storage file corresponding to the target storage server according to the storage file name and storage path. The storage file is a file with a preset file size created by the target storage server. The absolute offset is determined according to the actual offset of the storage file and the preset file size. Wherein, the absolute offset = offset - size, offset is the offset in append access, size is the preset file size, and the opening offset of the file stream storing the file is set to size-1.
2. The method according to claim 1, characterized in that, After determining whether the storage file corresponding to the target file has already been created on the target storage server, the method further includes: If no storage file corresponding to the target file is created in the target storage server, the file information of the target file is read, the file information includes basic file information and / or file data information, and the storage space required for the target file is determined based on the file information; A first instruction message is sent to the target storage server. The first instruction message carries the storage space information required by the target file, so that the target storage server can create a storage file based on the first instruction message and determine the storage file name and storage path corresponding to the storage file.
3. The method according to claim 2, characterized in that, Before transferring the file fragments to the target storage server based on the SFTP session channel, the process also includes: The system locks the current file creation permissions based on a global lock control and synchronizes the corresponding permission lock information with the target storage server, so that the target storage server creates a storage file based on the instruction message when creating a storage file.
4. The method according to claim 3, characterized in that, Also includes: In response to the completion of the creation of the storage file on the target storage server, the current file creation permission is unlocked, and the fragment write permission of the storage file is locked, so that when the file fragment is written to the storage file corresponding to the target storage server, the file fragment is written to the storage file only once.
5. The method according to claim 1, characterized in that, The file fragment transfer request is an HTTP PUT request, and the method further includes: When the information digest value of the file fragment is inconsistent with the preset information digest value, a second indication information is sent to the requesting end that initiated the file fragment transmission request. The second indication information is used to instruct the requesting end to resend the file fragment transmission request for the corresponding file fragment. The resent fragment transmission request carries the fragment update file corresponding to the file fragment.
6. A file fragmentation transmission device, characterized in that, include: The channel establishment module is configured to establish an SFTP session channel with the target storage server in response to a file fragment transfer request for the target file. The verification module is configured to verify whether the message digest value of each file fragment corresponding to the target file is consistent with a preset message digest value for each received file fragment. The transmission module is configured to transmit the file fragment to the target storage server based on the SFTP session channel when the information digest value of the file fragment is consistent with the preset information digest value. The shutdown module is configured to, in response to the completion of the transmission of all file fragments of the target file, when the total first message digest value of all file fragments matches the total second message digest value of the target file, combine the file fragments in the target storage server to obtain the target file, and close the SFTP session channel. The judgment module is configured to determine whether a storage file corresponding to the target file has been created in the target storage server. The transmission module includes: The acquisition unit is configured to acquire the storage file name and storage path corresponding to the storage file when a storage file corresponding to the target file has been created in the target storage server. The writing unit is configured to write the file fragments to the absolute offset of the storage file corresponding to the target storage server based on the SFTP session channel, according to the storage file name and storage path. The storage file is a file with a preset file size created by the target storage server. The absolute offset is determined based on the actual offset of the storage file and the preset file size. Wherein, the absolute offset = offset - size, offset is the offset in append access, size is the preset file size, and the opening offset of the file stream storing the file is set to size-1.
7. The apparatus according to claim 6, characterized in that, Also includes: The reading module is configured to read the file information of the target file when no storage file corresponding to the target file is created in the target storage server. The file information includes basic file information and / or file data information, and the module determines the storage space required for the target file based on the file information. The first instruction module is configured to send a first instruction message to the target storage server. The first instruction message carries storage space information required by the target file, so that the target storage server creates a storage file based on the first instruction message and determines the storage file name and storage path corresponding to the storage file.
8. The apparatus according to claim 6 or 7, characterized in that, The storage file is a file with a preset file size created by the target storage server; The writing unit is specifically configured to, based on the SFTP session channel, write the file fragments to the absolute offset of the storage file corresponding to the target storage server according to the storage file name and storage path, wherein the absolute offset is determined according to the actual offset of the storage file and the preset file size.
9. A server, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory, causing the server to perform the file fragmentation transmission method according to any one of claims 1-5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the file fragmentation transmission method as described in any one of claims 1-5.
11. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the file fragmentation transmission method as described in any one of claims 1-5.