File transmission method and system, electronic equipment and storage medium

Abstract

The invention discloses a file transmission method and system, electronic equipment and a storage medium. The method is applied to a server and can comprise the steps that in response to a file uploading request from a client side, a fragmentation strategy matched with a target file is sent to the client side, the file uploading request is used for requesting to upload the target file to the server, and the fragmentation strategy is used for representing that the client side carries out fragmentation processing on the target file; obtaining rules of a plurality of file fragments; obtaining a plurality of file fragments uploaded by the client; and writing the obtained file fragments into a storage space in a server. The technical problem that files cannot be effectively transmitted is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a file transmission method, system, electronic device, and storage medium. Background Art

[0002] Currently, when transferring files from a local client to a remote storage medium on a server, if the files to be transferred are large, such as terabyte (TB or T)-level files, they are subject to a series of limitations, such as memory limitations on the client or server. Such files can be directly compressed or encrypted for file transfer. This process can easily strain the performance of the server or client, requiring them to achieve an extremely high performance advantage. This can easily lead to server or client failures, resulting in unavailability of the corresponding service. Therefore, the technical problem of ineffective file transfer still exists.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a file transmission method, system, electronic device and storage medium to at least solve the technical problem of being unable to effectively transmit files.

[0005] According to one aspect of an embodiment of the present application, a file transmission method is provided. The method is applied to a server. The method may include: in response to a file upload request from a client, sending a sharding strategy matching a target file to the client, wherein the file upload request is for requesting upload of the target file to the server, and the sharding strategy represents a rule for the client to shard the target file to obtain multiple file shards; obtaining the multiple file shards uploaded by the client; and writing the obtained file shards to storage space on the server.

[0006] According to another aspect of an embodiment of the present application, another file transmission method is provided. The method is applied to a client. The method may include: sending a file upload request to a server, wherein the file upload request is used to request uploading of a target file to the server; receiving a sharding strategy matching the target file and sent by the server in response to the file upload request, wherein the sharding strategy is used to represent a rule for sharding the target file; sharding the target file according to the sharding strategy to obtain multiple file shards; and uploading the multiple file shards to the server, wherein the server stores the file shards in a storage space.

[0007] According to another aspect of an embodiment of the present application, a file transmission system is provided. The system may include: a client and a server, wherein the client is configured to send a file upload request to the server, wherein the file upload request is used to request that a target file be uploaded to the server; the server is configured to, in response to the file upload request, send a sharding strategy matching the target file to the client, wherein the sharding strategy is used to indicate a rule for the client to shard the target file to obtain multiple file shards; obtain the multiple file shards uploaded by the client; and write the obtained file shards to storage space on the server.

[0008] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device may include a memory and a processor: the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the file transmission method of the embodiment of the present application is implemented.

[0009] According to another aspect of an embodiment of the present application, a processor is further provided, wherein the processor is configured to run a program, wherein the file transmission method of the embodiment of the present application is executed when the program is running.

[0010] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the file transmission method of the present invention.

[0011] According to another aspect of the embodiments of the present application, a computer program product is provided, which includes a computer program that, when executed by a processor, implements the file transmission method of the embodiments of the present application.

[0012] In an embodiment of the present application, if the client needs to transfer a target file to the storage space of the server, a corresponding file upload request can be triggered and the request can be sent to the server. After the server detects the file upload request, it can determine a sharding strategy that matches the target file and send it to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transferred according to the sharding strategy to obtain multiple file shards. The client can send the file shards to the server to write them to the storage space in the server. Since the embodiment of the present application takes into account that if a file is transferred by a single encryption or compression method, it will lead to a situation where performance is fully utilized, a larger target file can be sharded to obtain multiple file shards, and each file shard can be encrypted, compressed, etc. respectively. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of full performance, thereby achieving the ability to meet the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transferring files and solving the technical problem of being unable to effectively transfer files.

[0013] It is easy to notice that the above general description and the following detailed description are merely for the purpose of exemplifying and explaining the present application, and do not constitute a limitation to the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 This is a schematic diagram of an application scenario of a file transmission method according to an embodiment of the present application;

[0016] Figure 2 is a flowchart of a file transmission method according to an embodiment of the present application;

[0017] Figure 3 is a flowchart of another file transmission method according to an embodiment of the present application;

[0018] Figure 4 is a flow chart of a file transmission system according to an embodiment of the present application;

[0019] Figure 5 is a schematic diagram of an efficient and stable transmission architecture for very large files according to an embodiment of the present application;

[0020] Figure 6is a flowchart of a file transfer request generation, transmission and verification process according to an embodiment of the present application;

[0021] Figure 7 is a flowchart of a process of transmitting a target file under different circumstances according to an embodiment of the present application;

[0022] Figure 8 This is a flowchart of an execution process when file segmentation is incomplete and file transfer is complete according to an embodiment of the present application;

[0023] Figure 9 This is a flowchart of a solution to a file fragment upload failure according to an embodiment of the present application;

[0024] Figure 10 This is a schematic diagram of an end-to-end interaction for efficient and stable transmission of an extremely large file according to an embodiment of the present application;

[0025] Figure 11 is a schematic diagram of a file transmission device according to an embodiment of the present application;

[0026] Figure 12 is a schematic diagram of another file transmission device according to an embodiment of the present application;

[0027] Figure 13 is a structural block diagram of a computer terminal according to an embodiment of the present application;

[0028] Figure 14 is a block diagram of an electronic device according to a file transmission method of an embodiment of the present application;

[0029] Figure 15 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a file transmission method according to an embodiment of the present application;

[0030] Figure 16 It is a structural block diagram of a computing environment of a file transmission method according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Example 1

[0034] According to an embodiment of the present application, a file transmission method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0035] The file transmission method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to this. Figure 1 In the application scenario shown, the server 10 can be a cloud. The server 10 can be connected to one or more client devices 20 via a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here can include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client devices together constitute the client relative to the server. An interactive interface for obtaining file upload requests can be deployed on the graphical user interface of the client device. The interactive interface can be a generative dialogue interface. The client device 20 can interact with the user through the graphical user interface to implement the file transmission method provided in the embodiment of the present application.

[0036] In an embodiment of the present application, the system composed of a client device and a server can perform the following steps: if the client needs to transfer a certain target file to the server, the file upload request corresponding to the target file to be uploaded can be input on the interactive interface of the client device, and can be sent to the server through the network. After receiving the file upload request, the server can perform the following steps: step S102, in response to the file upload request from the client, the sharding strategy matching the target file is sent to the client, wherein the file upload request is used to request to upload the target file to the server, and the sharding strategy is used to indicate the rule for the client to perform sharding processing on the target file and obtain multiple file shards; step S104, obtain the multiple file shards uploaded by the client; step S106, write the obtained file shards to the storage space in the server. In the above process, the sharding strategy determined by the server and the prompt information indicating whether the storage space is successfully written can be sent to the client device through the network. The corresponding sharding strategy and prompt information can be displayed on the interactive interface of the client device.

[0037] Since the embodiments of the present application take into account that file transfer through a single encryption or compression method will lead to performance bottleneck, a larger target file can be fragmented to obtain multiple file fragments, and each file fragment can be encrypted, compressed, etc. separately. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of performance bottleneck, thereby meeting the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transmitting files and solving the technical problem of being unable to effectively transmit files.

[0038] The embodiment of the present application proposes the following method from the technical implementation side. In the above application scenario, the present application provides the following Figure 2 The file transmission method shown can be applied in a server. Figure 2 is a flowchart of a file transmission method according to an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps:

[0039] Step S202, in response to a file upload request from the client, a sharding strategy matching the target file is sent to the client, wherein the file upload request is used to request uploading the target file to the server, and the sharding strategy is used to indicate that the client performs sharding processing on the target file to obtain a rule for multiple file shards.

[0040] In the technical solution provided in step S202 of the present application, upon detecting receipt of a file upload request from a client, a sharding strategy matching the target file to be uploaded in the file upload request can be determined, and the sharding strategy can be sent to the client. The file upload request is used to request upload of the target file to the server. The target file can be a file in the client, i.e., a local file on the client, or a relatively large file, such as a terabyte-level file, also known as an oversized file or a large file. The client can deploy a corresponding application to upload the target file to the storage space on the server, i.e., the client can be an application (APP) at the user layer. The client can also deploy a corresponding browser to upload the target file to the storage space on the server, i.e., the client can be a browser at the user layer. The server can be in the service layer and can perform self-developed transmission of the target file. The sharding strategy can be used to indicate the rules for the client to shard the target file to obtain multiple file shards, which can also be called a sharding rule or a transmission strategy. The file shards can be the sharding information of the target file after sharding. A file fragment may also be called a fragment file, and may be a file fragment stream data.

[0041] Optionally, the user layer supports multiple systems, such as Microsoft Windows (Windows), Linux, and CentOS. The user layer supports multiple sources, such as local / mobile hard drives and Network Attached Storage (NAS). The browser or APP used for file transfer in the client in the user layer can perform operations such as single file upload, folder upload, file download, folder download, upload pause or download pause, upload progress or download progress, upload list or download list, task cancellation (e.g., canceling the upload of the target file), task continuation (e.g., continuing the upload of the target file), viewing images, file compression, and file encryption.

[0042] Optionally, the service layer can perform tenant (client user) isolation, flow control, user verification, permission verification, adaptive fragmentation, instant transmission, fragment upload, breakpoint resumption, file download, file continuation, fragment MD5 verification / fragment overall verification (Total Check), large file streaming Secure Hash Algorithm (SHA) 1, file decryption and operation logs, etc. during the self-developed transmission process. Among them, MD5 verification can be a hash algorithm to verify the integrity of the file to ensure that the data has not been tampered with or damaged during transmission or storage. sha1 is a cryptographic hash function widely used in hash algorithms, also known as SHA1 or sha-1. It can convert data of any length, such as a file, into a fixed-length hash value, such as a fixed length of 160 bits, that is, convert the file into a 20-byte hash value. The hash value can be used to verify the integrity of the file, thereby preventing errors during file transmission.

[0043] It should be noted that the operations that can be performed by the above-mentioned user layer and service layer during the file transfer process are only examples and are not specifically limited here.

[0044] Optionally, if a user needs to transfer a specific file, they can select that file as the target file and generate a corresponding file upload request by clicking on the target file and selecting the upload action, or by dragging the target file to the upload area, etc., confirming with the server that they need to upload the target file and then sending the file upload request to the server. It should be noted that the above method and process for generating a file upload request after selecting a target file is for illustrative purposes only and is not intended to be a specific limitation.

[0045] Optionally, the client can transmit the file upload request initiated by the user to the server via a Hypertext Transfer Protocol Stream (HTTP stream). After the server receives the file upload request sent by a connected client, in order to ensure the security of the file upload and the security of the client, the file upload request can be verified to determine whether the server allows the target file corresponding to the file upload request to be uploaded, wherein the verification result can be used to indicate whether the server allows the client to upload the target file indicated by the file upload request. It should be noted that the above verification of the file upload request is only for illustration purposes. It is not necessary to verify the file upload request. The above verification process can also be used to determine whether the client that issued the file upload request has the authority to upload files, etc., to ensure the security of the file upload. No specific restrictions are made here.

[0046] Optionally, after the server allows the target file to be uploaded, a random key may be generated, or a sharding strategy corresponding to the target file may be determined, and the key and corresponding sharding strategy may be returned to the client.

[0047] Optionally, after the client obtains the encryption key and sharding strategy, it can call the sharding strategy to perform processing operations such as sharding, compression, and encryption on the file.

[0048] Because the related art performs encryption, compression, and other operations on the uploaded file to transfer it to the server, if the uploaded file is too large, this process will result in performance bottlenecks, and if the file transfer process fails, the entire file needs to be re-uploaded. Therefore, there is still a technical problem of improving the efficiency of file transfer. However, in the embodiment of the present application, the file can be automatically segmented, support point-to-point retransmission, and support integrity verification, thereby avoiding the situation of retransmission after file transfer failure. Based on streaming segment encryption and dynamic random key generation, the security of file transfer can also be guaranteed, thereby achieving the technical effect of improving the efficiency of file transfer.

[0049] Step S204: Acquire multiple file segments uploaded by the client.

[0050] In the technical solution provided in the above step S204 of the present application, after the dispatching strategy matching the target file is sent to the client, and the client segments the target file to obtain file segments, multiple file segments uploaded by the client can be obtained.

[0051] Optionally, after the client performs fragmentation processing on the target file according to a corresponding fragmentation strategy to obtain multiple file fragments, the file fragments can be transmitted to the server in a certain order.

[0052] For example, multiple file segments can be numbered, and when uploading the file segments, the file segments can be uploaded in ascending order of number. It should be noted that the above order and process of uploading multiple file segments to the client are merely examples and are not specifically limited here.

[0053] For another example, a stack can be set up to store file fragments. During the fragmentation process of the target file, the obtained file fragments can be pushed into the stack in a certain order. After the fragmentation process is completed, the file fragments in the stack can be transferred to the server in sequence by popping them out of the stack.

[0054] It should be noted that the above-mentioned process and method of uploading a file in segments to a server by a client, and the method of sequentially uploading each file segment, are merely illustrative and are not specifically limited herein. Any process and method for uploading multiple segments to a server that requires segmenting a file is within the scope of protection of the embodiments of this application.

[0055] Step S206: Write the acquired file segments into the storage space in the server.

[0056] In the technical solution provided in step S206 above, after obtaining the multiple file segments uploaded by the client, the obtained file segments can be written to the storage space in the server, where the storage space in the server is a disk in the server, which can also be called a (remote) storage medium. The storage medium can include an Object Storage Service (OSS), NAS, and a Cloud-Native Distributed File System (CPFS).

[0057] Optionally, after the server receives the multiple file segments uploaded by the client, it may perform corresponding decryption, decompression, etc. on the multiple file segments, and may write each file segment to a corresponding location in the storage space in the order of each file segment.

[0058] In the embodiment of the present application, the transmission of ultra-large text is mainly used to solve the problem of transferring local ultra-large files to remote storage media, for example, the transmission of ultra-large data of a large language model (LLM) or other machine learning models, to ensure the stability, reliability and consistency of file transmission. The embodiment of the present application can also automatically identify the performance of the server, network conditions, etc. to automatically optimize the transmission strategy, thereby reducing the performance consumption in the process of transferring files and ensuring the utilization of bandwidth. Based on file segmentation, it adopts distributed services, supports ultra-high concurrency, and has stable and reliable performance. Through the above operations, the technical effect of effectively transmitting files is achieved.

[0059] Through the above steps S202 to S206 of the present application, if the client needs to transfer the target file to the server's storage space, a corresponding file upload request can be triggered and sent to the server. After the server detects the file upload request, it can determine a sharding strategy that matches the target file and send it to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transferred according to the sharding strategy to obtain multiple file shards. The client can send the file shards to the server to be written to the storage space in the server. Since the embodiment of the present application takes into account that file transfer through a single encryption or compression method will lead to a performance hit, it can be achieved by sharding a larger target file to obtain multiple file shards. Each file shard can be encrypted, compressed, etc. separately. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of performance hit, thereby achieving the ability to meet the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transferring files and solving the technical problem of being unable to effectively transfer files.

[0060] The above method of this embodiment is further introduced below.

[0061] As an optional implementation, the method further includes: if the file attribute of the target file is greater than a file attribute threshold, obtaining a sharding strategy that matches the file attribute of the target file, wherein the file attribute is used to indicate the size of the target file.

[0062] In this embodiment, the file attributes of the target file may be detected. If it is detected that the file attributes are greater than a file threshold, a sharding strategy that matches the file attributes of the target file may be obtained. The file attributes may be used to indicate the size level of the target file. For example, the target file may be a file of byte (B), kilobyte (KB), megabyte (MB), gigabyte (GB), or T level. The file attribute threshold may be used to indicate that the file size level is T level.

[0063] Optionally, if a file upload request from the client is detected and the file upload request verification is passed, the file attributes of the target file can be determined. If the file attributes are less than the file attribute threshold, it can be said that the file is not an oversized file. At this time, there is no need to segment the target file, and the file can be transferred to the server through ordinary compression, encryption and other processes.

[0064] Optionally, if it is determined that the file attribute of the target file is greater than or equal to the file attribute threshold, it can be said that the file is an oversized file. If it is directly transmitted to the server through the ordinary compression and encryption process, there will be situations such as full performance, and it will be subject to the memory limitations of the server and the client. In addition, this method cannot be transmitted concurrently, and when the transmission fails, the entire file needs to be retransmitted, and there is still a situation of low transmission efficiency. Based on the above situation, in an embodiment of the present application, it can be considered to fragment such oversized files with file attributes greater than or equal to the file attribute threshold, and perform compression, encryption and other processes on each file fragment separately to achieve concurrency. If it is detected that a certain file fragment has not been transmitted to the server, only the file fragment needs to be transmitted, rather than the entire file itself, thereby achieving the technical effect of improving the efficiency of file transmission.

[0065] As an optional implementation, if the file attributes of the target file are greater than the file attribute threshold, a sharding strategy that matches the file attributes of the target file is obtained, including: determining the operating system on which the client is running; if the file attributes of the target file are greater than the file attribute threshold, a sharding strategy that matches both the operating system and the file attributes is obtained.

[0066] In this embodiment, in the process of determining that the file attribute of the target file is greater than the file attribute threshold and obtaining a sharding strategy that matches the file attribute of the target file, the operating system run by the client can be determined. If the file attribute of the target file is greater than the file attribute threshold, a sharding strategy that matches both the operating system and the file attribute can be determined. The operating system can be Windows, Linux / CentOS, etc., or a file system such as File Allocation Table 32 (FAT32), New Technology File System (NTFS), and Extended File Allocation Table (exFAT). This is only for illustration and no specific limitation is imposed.

[0067] Alternatively, due to file size limitations in different operating systems, for example, the maximum file size supported by the FAT32 file system is 4 gigabytes (GB), while NTFS can support larger file sizes. The performance of different operating systems can also be considered. Since the target file runs on an operating system, the processing power and memory limitations of the operating system need to be considered to determine the appropriate shard size. Taking into account the above two situations, it is necessary to match the sharding strategies of different operating systems with respect to the file sizes supported by the above-mentioned different operating systems and the processing power of different operating systems. That is, according to the file sizes supported by different operating systems and the size of the processing power, the target file is fragmented into shard files that meet the corresponding file sizes and the processing power of the operating system.

[0068] For example, you can also consider network transmission or server storage device limitations. If the target file needs to be transmitted over the network or stored on a specific storage device on the server, you need to consider the network bandwidth, transmission speed, or file size limitations of the storage device. Taking these factors into account, you can determine a sharding strategy that is consistent with the current network bandwidth and transmission efficiency.

[0069] It should be noted that the above-mentioned method and process of formulating a sharding strategy are only for illustration and are not specifically limited here. As long as the corresponding sharding strategy can be adaptively formulated according to the actual situation, it is within the scope of protection of the embodiments of this application.

[0070] When the file size of the target file exceeds the T level, if only a single fragmentation strategy is used to fragment the target file without considering the operating system, network speed, etc., it may cause the fragmented files obtained according to the single fragmentation strategy to be affected by the network speed during transmission and unable to be transmitted, or the file size of the fragmented files cannot be supported by the operating system. Therefore, there is still a technical problem of low accuracy in formulating fragmentation strategies. However, in an embodiment of the present application, a corresponding fragmentation strategy can be adaptively obtained according to the file size restrictions of different operating systems, the network conditions during the current transmission process, etc., so that the fragmented files obtained by fragmentation can comply with the current network conditions and be within the file size restrictions of the operating system, thereby achieving the technical effect of improving the accuracy of formulating fragmentation strategies.

[0071] As an optional implementation, the method also includes: obtaining an encryption key and an upload identifier corresponding to the target file, wherein the encryption key is used to encrypt the target file, and the upload identifier is used to identify the upload process of the target file; storing the encryption key and the upload identifier; and obtaining a sharding strategy when the encryption key and the upload identifier are successfully stored.

[0072] In this embodiment, the encryption key and upload identifier corresponding to the target file can be obtained, stored, and the corresponding sharding strategy can be obtained. The encryption key is used to provide to the client to encrypt the target file and can be a random (encryption) key. The upload identifier can be used to identify the upload process of the target file and can be an uploadID.

[0073] Optionally, when performing multi-part uploads (for example, uploading large files), a unique identifier is required to identify the upload operation. When performing multi-part uploads, an upload task is first initialized, which generates an upload task unique identifier (uploadIdentity Document, abbreviated as uploadID). Each upload request for the target file's file segments must carry this uploadID to identify them as belonging to the same upload task. The uploadID allows the server to merge file segments belonging to the same target file, ultimately completing the file upload.

[0074] Optionally, if the target file is allowed to be uploaded, a random key may be generated, or an uploadID corresponding to the upload task for uploading the target file may be generated.

[0075] Optionally, the random key and uploadID are stored in an in-memory database (redis) in the server. For example, the random key and uploadID can be stored as (key; uploadID) in redis.

[0076] Optionally, if the storage is successful, the memory database can send a corresponding prompt message to the server to prompt that the storage is successful and the next step can be carried out, that is, the corresponding sharding rules can be obtained from the server.

[0077] As an optional implementation, the method also includes: when the encryption key and upload identifier are successfully stored, sending the encryption key and upload identifier to the client; obtaining multiple file fragments uploaded by the client, including: obtaining multiple encrypted file fragments uploaded by the client according to the upload identifier, wherein the encrypted file fragments are obtained by the client encrypting the file fragments according to the encryption key; calling the stored encryption key according to the upload identifier, decrypting the encrypted file fragments, and obtaining the file fragments.

[0078] In this embodiment, if the encryption key and upload identifier are successfully stored, the encryption key and upload identifier can be sent to the client. In the process of obtaining multiple file segments uploaded by the client, the client can upload multiple encrypted file segments according to the upload identifier, and the stored encryption key can be called according to the upload identifier to decrypt the encrypted file segments to obtain the corresponding file segments.

[0079] Optionally, after determining the encryption key and upload identifier for the target file currently to be uploaded and successfully storing both, the encryption key and upload identifier can be sent to the client, that is, the uploadID+random encryption key+sharding rule can be returned to the client.

[0080] Optionally, after the client successfully receives the uploadID, random encryption key, and fragmentation rule sent by the server, it can fragment the target file according to the fragmentation rule to obtain multiple file fragments, i.e., multiple fragment information to be uploaded. The file fragments can also be encrypted using the received random encryption key. Alternatively, the uploadID can be added to each fragment file.

[0081] For example, the fragment information to be uploaded can be pushed into a stack, where the stack can be represented as Stack<chunkidx,retryCount> , where chunkidx can be used to represent the stack index information, that is, the stack block index, which can be used to locate the stack where the shard information is inserted. The number of retries (retryCount) can be used to indicate whether there is an exception when the shard information is pushed into the stack, and the number of times the shard information is re-pushed into the stack if an exception occurs.

[0082] Optionally, after each file segment is pushed onto the stack based on the above operation, the file segment can be extracted from the stack and uploaded to the server until all the file segments written into the stack have been uploaded. Optionally, each file segment can be uploaded to the server in sequence by popping the file segment from the stack.

[0083] In the embodiments of the present application, the technical effect of security in the file transmission process can be achieved through streaming segment encryption algorithms, dynamic keys, etc.

[0084] As an optional implementation, the encrypted file segments are compressed by the client, and the compression results are encrypted according to the encryption key, wherein the stored encryption key is called according to the upload identifier, and the encrypted file segments are decrypted to obtain the file segments, including: calling the stored encryption key according to the upload identifier, decrypting the encrypted file segments to obtain the compression results; and decompressing the compression results to obtain the file segments.

[0085] In this embodiment, the stored encryption key is called according to the upload identifier to decrypt the encrypted file segments. In the process of obtaining the decrypted file segments, the stored encryption key can be called according to the upload identifier to decrypt the encrypted file segments to obtain the compressed result. The compressed result can be decompressed to obtain the file segments, wherein the encrypted file segments can be obtained by compressing the file segments by the client and encrypting the compressed result according to the encryption key.

[0086] Optionally, when the server receives each file segment, it can determine whether the file segment is a file segment of the target file to be uploaded by determining whether there is an uploadID in each file segment. If there is an uploadID in the file segment, the server can call the secret key pre-stored in redis to decrypt the file segment to obtain the decrypted file segment.

[0087] Optionally, after decrypting each encrypted file segment, a compressed file segment in a decrypted state, i.e., a compression result, can be obtained. Each compressed file segment after decompression can be decompressed to obtain each file segment of the target file, i.e., a file stream of the target file.

[0088] Optionally, since a large number of files have different file types, a specific compression technology may be used to compress each file segment of a target file of a different file type.

[0089] As an optional implementation, step S208, writing the obtained file fragments into the storage space in the server, includes: determining the data offset of the obtained file fragments in the target file, wherein the data offset is used to indicate the position of the file fragments in the target file; based on the data offset, writing the file fragments into the storage space.

[0090] In this embodiment, in the process of writing the obtained file fragments into the storage space in the server, the data offset of the obtained file fragments in the target file can be determined, and the file fragments can be written into the storage space according to the data offset, wherein the data offset can be used to indicate the position of the file fragment in the target file, and can also be called the fragment offset and offset position.

[0091] Optionally, since the position of each file slice in the target file is different when the target file is divided into multiple file slices for transmission, the server needs to calculate the data offset of each file slice based on the information in the request message to ensure that each file slice can be correctly written to the corresponding position of the target file. In this way, after receiving the various file slices of the target file, the server can correctly reassemble the slice data into the target file based on these data offsets.

[0092] Optionally, after the server obtains each file segment, it may calculate an offset position of each file segment, and write the file segment into the storage space according to the offset position.

[0093] As an optional implementation, writing the file shard into the storage space based on the data offset includes: determining, according to the data offset, mapping the file content of the file shard to the write position in the storage space; and writing the file content of the file shard to the write position.

[0094] In this embodiment, in the process of writing the file segment to the storage space based on the data offset, the write position mapped to the storage space can be determined according to the data offset, and the file content of the file segment can be mapped to the write position of the storage space.

[0095] Optionally, after calculating the data offset of the file shard through the request message, it can be memory-mapped and written to the storage space of the server, and the write position of each file shard stored in the storage space can be recorded, that is, the location of the file shard can be recorded.

[0096] For example, the offset position of the file fragment stream data can be calculated based on the request message. It can be written to the server disk through the memory mapping of New Input / Output (nio) and the position is recorded. If all the positions (file fragments) are transferred, the next step can be carried out.

[0097] For another example, the completion of the transmission of each file segment can be indicated by a binary bit. If there are five file segments to be transmitted, the initial bit state is 00000. If the first file segment is transmitted, the bit state can be adjusted to 10000. If the first and fourth file segments are transmitted, the bit state can be adjusted to 10010. It should be noted that the above method and process of marking whether the file segment transmission is complete are for illustration only and are not specifically limited here.

[0098] It should be noted that the memory mapping method used above is only an example and is not specifically limited here. As long as the method and process can write file segment stream data to the storage space of the server, it is within the scope of protection of the embodiments of this application.

[0099] As an optional implementation, step S208, writing the obtained file shards into the storage space in the server, includes: determining a first hash value of the obtained file shards on the server, wherein the first hash value is used to represent the file content of the file shards on the server; if the first hash value is the same as the second hash value of the file shards on the client, writing the obtained file shards into the storage space in the server, wherein the second hash value is used to represent the file content of the file shards on the client.

[0100] In this embodiment, in the process of writing the obtained file shards to the storage space in the server, the first hash value of the obtained file shards on the server can be determined, and the second hash value of the file shards on the client can also be determined. The relationship between the first hash value and the second hash value can be determined. If the first hash value and the second hash value are the same, the obtained file shards can be written to the storage space in the server, wherein the first hash value can be used to represent the file content of the file shards on the server, and the second hash value can be used to represent the file content of the file shards on the client.

[0101] Optionally, after the server obtains the file segment from the client, it can determine from Redis whether the first hash value (sha1) corresponding to the received file segment exists. If so, Redis can return the data to the server, that is, send the first hash value to the server. At this time, the server can return a corresponding prompt message to the client to prompt the client that the file segment exists, that is, the file segment has been uploaded successfully, and the upload process of other file segments can continue.

[0102] Optionally, after all file segments of the entire target file are uploaded and decrypted and decompressed, the first hash value obtained above and the second hash value in the client contained in the file segment itself can be used to determine whether the uploaded file is consistent with the file to be uploaded.

[0103] Optionally, if the first hash value and the second hash value are the same, it can be said that the corresponding file segment is consistent with the file segment to be transmitted in the client; if the first hash value and the second hash value are different, it can be said that the corresponding file segment is different from the file segment to be transmitted by the client. If the first hash value of each file segment of the target file received by the server is the same as the corresponding second hash value, it can be said that the target file received by the server is the same as the target file to be transmitted by the client. Conversely, if the first hash value of at least one file segment in the target file received by the server is different from the corresponding second hash value, it can be said that the target file received by the server is different from the target file to be transmitted by the client.

[0104] Optionally, if the first hash value and the second hash value of a certain file segment are different, the server may send corresponding information to the client to prompt the client to resend the file segment to the server.

[0105] For example, before the client transmits the target file, the target file can be fragmented according to the received fragmentation strategy to obtain file fragments. After obtaining the file fragments, the sha1 value of each file fragment can be calculated once. After that, the file fragments can be transmitted to the server. After receiving each file fragment, the server can calculate the sha1 value of each received file fragment once. It can determine whether the sha1 value of each file fragment in the client is equal to the sha1 value of the corresponding file fragment in the server through fragmentation to determine whether the target file is transmitted correctly. That is, if there is at least one pair of sha1 values ​​that are not equal, it is determined that the target file is transmitted incorrectly. Conversely, if the sha1 values ​​of each fragment file of the target file are equal, it is determined that the target file is transmitted correctly.

[0106] Considering the problem in the related art that if a file is transferred incorrectly or fails, the entire file needs to be re-uploaded, in the embodiment of the present application, it is possible to automatically segment very large files and support integrity verification and breakpoint resuming. That is, integrity verification can be used to determine whether each file segment received by the server is consistent with each file segment to be transmitted by the client to determine whether the transmission is successful. If they are inconsistent, the inconsistent file segments can be retransmitted, that is, breakpoint resuming. The above method can solve the problem of retransmitting the entire file when the transfer of very large text fails, thereby achieving the purpose of transmitting only the file segment that failed to be transferred, thereby achieving the technical effect of improving the efficiency of file transfer.

[0107] As an optional implementation, step S202, in response to a file upload request from the client, sends a sharding strategy that matches the target file to the client, including: in response to the file upload request from the client, verifying the file upload request to obtain a verification result; if the verification result is that the server allows the client to upload the target file, sending a sharding strategy that matches the target file to the client.

[0108] In this embodiment, after detecting the client's file upload request, the file upload request can be verified to determine whether the verification result indicates that the server allows the client to upload the target file in the file upload request. If allowed, the sharding strategy matching the target file can be sent to the client, wherein the verification result can be used to indicate whether the server allows the client to upload the target file indicated by the file upload request.

[0109] Optionally, the client can transmit the file upload request initiated by the user to the server via an HTTP stream request. After the server receives the file upload request sent by a connected client, in order to ensure the security of the file upload and the security of the client, the file upload request can be verified to determine whether the server allows the target file corresponding to the file upload request to be uploaded. The verification result can be used to indicate whether the server allows the client to upload the target file indicated by the file upload request.

[0110] It should be noted that the above-mentioned verification of the file upload request is only for illustration purposes. It is not necessary to verify the file upload request. The above-mentioned verification process can also be used to determine whether the client issuing the file upload request has the authority to upload the file, etc., to ensure the security of the file upload. No specific restrictions are made here.

[0111] For example, after the server receives the corresponding file upload request, it can perform user verification and permission verification on the user of the client that issued the file upload request, that is, it can verify the user's login status and the user's resource permission operation verification. It can also verify the target file indicated in the issued file upload request, for example, it can determine whether there is a file with the same name as the target file. If the user verification and permission verification pass, and the target file does not have a file with the same name, it can be determined that the server allows the client to upload the target file. Conversely, if at least one of the user verification, permission verification, and the target file does not have a file with the same name is not satisfied, it can be determined that the server does not allow the client to upload the target file.

[0112] For example, after the server receives the corresponding file upload request, it can perform user verification and permission verification on the user of the client that issued the file upload request, that is, it can verify the user's login status and the user's resource permission operation verification. It can also verify the target file indicated in the issued file upload request, for example, it can determine whether there is a file with the same name as the target file. If the user verification and permission verification pass, and the target file does not have a file with the same name, it can be determined that the server allows the client to upload the target file. Conversely, if at least one of the user verification, permission verification, and the target file does not have a file with the same name is not satisfied, it can be determined that the server does not allow the client to upload the target file.

[0113] It should be noted that the above process and method for verifying file upload requests are only for illustration and are not specifically limited here.

[0114] Optionally, if the server verifies the user and the target file in the file upload request and both pass the verification, the server may allow the user's client to upload the target file. The server may determine the sharding rule that matches the target file.

[0115] As an optional implementation, step S202, in response to a file upload request from a client, verifies the file upload request to obtain a verification result, including: in response to a file upload request from a client, performs at least one of the following verifications on the file upload request to obtain a verification result: verifies the login status of the client's account in the file upload request; verifies the resource operation permissions associated with the client's account in the file upload request, wherein the resource operation permissions are used to limit the client's flow; verifies the target identification information of the target file in the file upload request, wherein the target identification information is used to represent the file name and file content of the target file.

[0116] In this embodiment, in the process of performing verification based on the file upload request from the client and obtaining the verification result, at least one of the following verifications can be performed to obtain the verification result: the login status of the client account in the file upload request can be verified. The resource operation authority associated with the client account in the file upload request can be verified. The target identification information of the target file summarized in the file upload request can also be verified, wherein the resource operation authority can be used to limit the client's flow, which can also be called resource authority operation verification. The target identification information can be used to represent the file name and file content of the target file, and the target identification information verification can also be a same name + sha1 verification.

[0117] Optionally, after receiving the file upload request from the client, the server can perform flow control. For example, the flow can be limited and controlled according to the dimension of organization (org). This flow control can limit the request flow of a specific organization to ensure the reasonable allocation of resources and the stability of the system. Through flow control, it is possible to avoid excessive requests from a certain organization causing excessive system load, or ensure that each organization can obtain fair resource allocation.

[0118] Optionally, the server may perform a user login status check on the user of the client that issued the file upload request according to the file upload request to determine whether the user has successfully logged in.

[0119] Optionally, the server may determine whether the user has permission to send file transfers based on the file upload request and according to a user resource permission operation check.

[0120] Optionally, the server can verify the target identification information based on the file upload request to determine whether there is a file with the same file name as the target file to be transferred in the current file upload request, that is, to determine whether there is a file with the same name as the target file. It can also determine whether there is a file with the same file content, that is, SHA1, as the target file to be transferred.

[0121] As an optional implementation, the method further includes: in response to the account login status verification being passed, the resource operation permission verification being passed, and the absence of identification information identical to the target identification information in the server, determining that the verification result is that the server allows the client to transfer the target file.

[0122] In this embodiment, it can be determined whether the account login status verification is passed. If passed, the resource operation permission verification can be passed, and there is no identification information identical to the target identification information in the server, to determine that the server allows the client to transfer the target file at this time.

[0123] Alternatively, if the account login status verification is confirmed to have passed, this indicates that the user's login status has been verified, meaning that the user's account is logged in on the client. If the resource operation permission verification is confirmed to have passed, this indicates that the user corresponding to the client has permission to request the server to transfer the file. If the server does not contain identification information identical to the target identification information, this indicates that the target file currently being transferred does not exist in the server's storage space. The next step in the normal file upload process can then proceed.

[0124] Optionally, if the server has identification information identical to the target identification information, that is, the server has the same file corresponding to the current target file, then a corresponding prompt message can be sent, such as direct idempotence, and returned to the client, where direct idempotence can be used to indicate direct return. Idempotence means that the effect of an operation being executed multiple times is the same as that of being executed once, that is, after executing the operation multiple times, the state of the server will not change.

[0125] As an optional implementation, the method further includes: adjusting the sharding strategy based on at least one of the following information: a performance indicator of the server, a performance indicator of the client, and a network status between the server and the client.

[0126] In this embodiment, the sharding strategy can be adjusted based on one of the following: server performance indicators, client performance indicators, and the network status between the server and the client. The server performance indicators and the client performance indicators can be collectively referred to as machine performance. The network status can also be referred to as network conditions.

[0127] In an embodiment of the present application, during the file transfer process, the machine performance and network conditions can be automatically identified to automatically optimize the sharding strategy, so that the transmission strategy can be adjusted according to the actual situation, making the file transfer process more accurate, thereby achieving the technical effect of effectively transmitting files.

[0128] As an optional implementation, the method further includes: in response to a failure in obtaining the file segments uploaded by the client, sending a retransmission request to the client, wherein the retransmission request is used to request the client to upload the file segments to the server again.

[0129] In this embodiment, if it is detected that the file segment uploaded by the client fails, a resend request may be sent to the client, wherein the resend request is used to request the client to upload the file segment to the server again.

[0130] Optionally, the server may detect the received file segments of the target file, for example, by checking whether the first hash value and the second hash value are the same to determine whether each file segment has been successfully uploaded. If the first hash value and the second hash value of a file segment are different, the server may send a corresponding retransmission request to the client, prompting the client to re-upload the file segment.

[0131] In the embodiment of the present application, considering the problem that if a file transmission error or transmission failure occurs, the entire file needs to be re-uploaded, it can be automatically fragmented by oversized files, and support integrity verification and breakpoint resumption. That is, integrity verification can be used to determine whether each file fragment received by the server is consistent with each file fragment to be transmitted by the client to determine whether the transmission is successful. If they are inconsistent, the inconsistent file fragments can be retransmitted, that is, breakpoint resumption. The above method can solve the problem of retransmitting the entire file when the oversized text transmission fails, thereby achieving the purpose of transmitting only a file fragment that failed to be transmitted, and further achieving the technical effect of improving the efficiency of file transmission.

[0132] Optionally, when determining that the segment upload has failed, it also includes situations such as server errors and network transmission errors. In these situations, the server can send a retransmission request (fai l) to the client, and the client can determine the number of times the above situation occurs, that is, the number of retries (retryCount), and determine the relationship between the number of retries and the retry threshold (maxretryCount). If the number of retries is greater than the retry threshold, that is, if (retCount>maxretryCount), the upload failure process can be followed.

[0133] Optionally, after entering the upload failure process, the client can rewrite the failed file segment to the stack. For example, the failed file segment can be re-stacked (pushStack) and the index information of the failed file segment in the stack (chunkidx) can be recorded. Since this is a retry upload process after confirming that the upload of the file segment has failed, the number of retries at this time needs to be increased by one (retryCount+1), that is, pushStack(chunkidx,retryCount+1), and the file segment is popped out of the stack and transmitted to the server.

[0134] Optionally, if the re-upload of the file segment is successful, the server can return a success message (success) to the client, for example, uploadRecord.done.push(chunkidx), uploadRecord.needUpload.remove(chunkidx), where uploadRecord can be used to represent the upload record of the operation of uploading the segmented file to the server. uploadRecord.needUpload.remove(chunkidx) can be used to indicate the removal of the chunkidx corresponding to the successfully uploaded file segment from the stack that needs to be uploaded to the server. uploadRecord.done.push(chunkidx) indicates that the chunkidx corresponding to the successfully uploaded file segment is added to the record of the completed upload.

[0135] As an optional implementation, after writing the acquired file segments into the storage space in the server, the method further includes: recording an operation log corresponding to the target file, wherein the operation log is used to indicate the operation process of uploading the target file from the client to the server.

[0136] In this embodiment, after the acquired file segments are written to the storage space in the server, the operation log of the target file can be recorded, wherein the operation log can be used to indicate the operation process of uploading the target file from the client to the server, which can also be called the operation flow.

[0137] Optionally, after the transmission of each file segment corresponding to the target file is completed, the operation flow can be recorded and the lock of the concurrent thread of the upload task can be released, that is, the lock is released and the related resources in the transmission process can be released.

[0138] The present application also provides a method for transmitting files from the client side. Figure 3 is a flowchart of a file transmission method according to an embodiment of the present application, such as Figure 3 As shown, the method is applied to the client and may include the following steps:

[0139] Step S302: Send a file upload request to the server, wherein the file upload request is used to request uploading a target file to the server.

[0140] In the technical solution provided in the above step S302 of the present application, if the user needs to transfer a target file, he can perform corresponding operations on the client to generate a file upload request, and send it to the server, where the file upload request can be used to request to upload the target file to the server.

[0141] Optionally, if the user needs to transfer a certain target file, the corresponding operation can be performed on the client. For example, the user can determine the file path (filePath) of the target file to be uploaded on the client, and call the mult ipartUpload function to divide the target file under the above file path into blocks and upload it. That is, if the user needs to upload the target file, the mult ipartUpload (filePath) operation can be triggered to generate a file upload request, wherein the mult ipartUpload function can use filePath as a parameter to represent the path of the target file to be uploaded in blocks, and divide the target file under the file path into multiple parts for uploading, thereby improving the upload speed and stability.

[0142] Optionally, since sha1 calculation is very time-consuming, if the size of the target file is large, it may cause the file upload request to time out (timeout). Therefore, if the size of the target file exceeds a certain threshold, for example, the size of the target file exceeds 50G, asynchronous sha1 calculation is adopted, that is, streaming sha-1 calculation is performed. After the client generates the file upload request, in order to avoid the situation where the file upload request times out, the transmission initialization can be performed, for example, beginUploadTransaction(orgid,path+filename,filesize,streamingsha-1(less than 50G, the last one is given for more than 50G)), thereby transmitting the above-mentioned file upload request to the server. After the user performs the corresponding operation and triggers multipartUpload(filePath) to indicate the target file that needs to be uploaded to the server, the transmission initialization can be started, that is, the start upload transaction (beginUploadTransaction) is started. You can first determine the identification information (organization identification, referred to as orgid) of the client that needs to upload the target file, and you can determine the file path and file name (path+filename) of the target file that the client needs to upload, as well as the file size (files size) of the target file. For files with a file size less than 50G, you can determine the calculation method of sha-1 at this time. For files with a file size greater than 50G, sha-1 is calculated asynchronously (streaming sha-1 (less than 50G, the last time is given for files above 50G)). It should be noted that if it is determined that the size of the target file exceeds 50G, then asynchronous sha1 calculation is performed for example only. There is no specific restriction on the file size for confirming whether to start the asynchronous sha1 algorithm. It can be flexibly adjusted according to actual conditions.

[0143] Step S304: receiving a fragmentation strategy that matches the target file and is sent by the server in response to the file upload request, wherein the fragmentation strategy is used to represent a rule for fragmenting the target file.

[0144] In the technical solution provided in the above step S304 of the present application, after sending a file upload request to the server, the server can determine whether the client is allowed to upload the target file. If allowed, the client can receive a sharding strategy that matches the target file sent by the server to the client, wherein the sharding strategy can be used to represent the rules for sharding the target file.

[0145] Optionally, if the server verifies the user and the target file in the file upload request and both pass the verification, the server may allow the user's client to upload the target file. The server may determine the sharding rule that matches the target file.

[0146] Optionally, after the server allows the target file to be uploaded, a random key may be generated, and a sharding strategy corresponding to the target file may be determined, and the key and corresponding sharding strategy may be returned to the client.

[0147] Step S306: Slice the target file according to the slicing strategy to obtain multiple file slicings.

[0148] In the technical solution provided in the above step S306 of the present application, after determining that the verification result is that the server allows the client to upload the target file and receiving the sharding strategy that matches the target file sent by the server, the target file can be sharded according to the sharding strategy to obtain multiple file shards.

[0149] Optionally, the client fragments the target file according to a corresponding fragmentation strategy to obtain multiple file fragments.

[0150] Optionally, after the client obtains the encryption key and sharding strategy, it can call the sharding strategy to perform processing operations such as sharding, compression, and encryption on the file.

[0151] Step S308: Upload the multiple file segments to the server, wherein the file segments are stored in the storage space by the server.

[0152] In the technical solution provided in the above step S308 of the present application, after the target file is fragmented according to the fragmentation strategy to obtain multiple file fragments, the multiple file fragments can be uploaded to the server, wherein the file fragments can be stored in the storage space by the server.

[0153] Optionally, after the client performs fragmentation processing on the target file according to a corresponding fragmentation strategy to obtain multiple file fragments, the file fragments can be transmitted to the server in a certain order.

[0154] For example, multiple file segments may be numbered respectively, and when uploading the file segments, they may be uploaded in order from smallest to largest number.

[0155] For another example, a stack for storing file fragments can be set up. During the fragmentation process of the target file, the obtained file fragments can be pushed into the stack in a certain order. After the fragmentation process is completed, the file fragments in the stack can be transferred to the server in sequence by popping them out of the stack.

[0156] Through the above steps S302 to S308 of the present application, a file upload request is sent to the server, wherein the file upload request is used to request uploading a target file to the server; a sharding strategy matching the target file and sent by the server in response to the file upload request is received, wherein the sharding strategy is used to represent the rules for sharding the target file; according to the sharding strategy, the target file is sharded to obtain multiple file shards; the multiple file shards are uploaded to the server, wherein the file shards are stored in the storage space by the server, thereby achieving the technical effect of effectively transmitting files and solving the technical problem of being unable to effectively transmit files.

[0157] The above method of this embodiment is further introduced below.

[0158] As an optional implementation, the method further includes: compressing the file segments to obtain compression results; and encrypting the compression results according to an encryption key from the server to obtain encrypted file segments.

[0159] In this embodiment, the file segments may be compressed to obtain a compression result, and the compression result may be encrypted according to an encryption key from a server to obtain an encrypted file segment.

[0160] Optionally, after determining the encryption key and upload identifier for the target file currently to be uploaded and successfully storing both, the encryption key and upload identifier can be sent to the client, that is, the uploadID+random encryption key+sharding rule can be returned to the client.

[0161] As an optional implementation, step S308, uploading multiple file segments to the server, includes: uploading the encrypted file segments to the server according to the upload identifier from the server, wherein the upload identifier is used to identify the upload process of the target file, the encrypted file segments are called by the server according to the upload identifier using the stored encryption key, the encrypted file segments are decrypted to obtain compression results, and the compression results are decompressed by the server to obtain file segments.

[0162] In this embodiment, in the process of uploading multiple file segments to the server, an upload identifier can be received from the server, and the encrypted file segments can be uploaded to the server, wherein the upload identifier can be used to identify the upload process of the target file. The encrypted file segments are called by the server according to the upload identifier. The stored encryption key is decrypted to obtain the compressed result, and the compressed result is decompressed by the server to obtain the file segments.

[0163] Optionally, after the client successfully receives the uploadID, random encryption key, and fragmentation rule sent by the server, it can fragment the target file according to the fragmentation rule to obtain multiple file fragments, i.e., multiple fragment information to be uploaded. The file fragments can also be encrypted using the received random encryption key. Alternatively, the uploadID can be added to each fragment file.

[0164] Optionally, based on the above operation, after each file segment is pushed into the stack, each file segment can be uploaded to the server. Each file segment can be uploaded to the server in a popping manner.

[0165] As an optional implementation, after uploading multiple file segments to the server, the method also includes: sending a file download request to the server, wherein the file download request is used to request the server to transmit the target file to the client; obtaining multiple file segments sent by the server in response to the file download request; and generating a target file from the multiple file segments.

[0166] In this embodiment, after uploading multiple file segments to the server, a file download request may be sent to the server, wherein the file download request may be used to request the server to transmit a target file to the client. Multiple file segments sent by the server in response to the file download request may be obtained, and the target file may be generated from the multiple file segments.

[0167] Optionally, if a user needs to download a target file already stored in the server, he needs to send a corresponding file download request to the server. After receiving the file download request, the server can call out the various file segments of the target file from the storage space and send them to the client in the corresponding order. The client can sort and combine the file segments in the corresponding order to obtain the required target file.

[0168] In an embodiment of the present application, if the client needs to transfer a target file to the storage space of the server, a corresponding file upload request can be triggered and the request can be sent to the server. After the server detects the file upload request, it can determine a sharding strategy that matches the target file and send it to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transferred according to the sharding strategy to obtain multiple file shards, and can also compress, encrypt, and other operations on the file shards. The client can send the file shards in a compressed, encrypted, and other state to the server. The server can decompress and decrypt the file shards and write them to the storage space in the server. Since the embodiments of the present application take into account that file transfer through a single encryption or compression method will lead to performance bottleneck, a larger target file can be fragmented to obtain multiple file fragments, and each file fragment can be encrypted, compressed, etc. separately. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of performance bottleneck, thereby meeting the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transmitting files and solving the technical problem of being unable to effectively transmit files.

[0169] Example 2

[0170] According to an embodiment of the present application, an embodiment of a file transmission system is also provided. Figure 4 is a schematic diagram of a file transmission system according to an embodiment of the present application, such as Figure 4 As shown, the file transmission system 400 may include: a client 401 and a server 402 .

[0171] The client 401 is used to send a file upload request to the server, wherein the file upload request is used to request to upload a target file to the server.

[0172] In this embodiment, the client 401 may send a file upload request to the server, wherein the file upload request may be used to request uploading a target file to the server 402 .

[0173] Optionally, if the user needs to upload a target file, the user may perform a corresponding operation to generate a file upload request, and may transmit the file upload request to the server 402 .

[0174] Server 402 is used to respond to a file upload request and send a sharding strategy that matches the target file to the client, where the sharding strategy is used to indicate the rules for the client to perform sharding processing on the target file to obtain multiple file shards; obtain multiple file shards uploaded by the client; and write the obtained file shards to the storage space in the server.

[0175] In this embodiment, after receiving the file upload request from the client 401, the server 402 can verify the file upload request. If the verification result shows that the server 402 allows the client 401 to upload the target file, the sharding strategy that matches the target file can be sent to the client. The server 402 can receive multiple file shards uploaded by the client 401 and write the file shards to the storage space in the server.

[0176] Optionally, if the server 402 verifies the user and the target file in the file upload request and both pass the verification, the server 402 may allow the user's client 401 to upload the target file. The server 402 may determine the fragmentation rule that matches the target file.

[0177] Optionally, after the server 402 allows the target file to be uploaded, a random key may be generated, or a sharding strategy corresponding to the target file may be determined, and the key and corresponding sharding strategy may be returned to the client 401 .

[0178] Optionally, after the client 401 obtains the encryption key and the fragmentation strategy, the fragmentation strategy can be called to perform processing operations such as fragmentation, compression, and encryption on the file.

[0179] Optionally, after the client 401 fragments the target file according to a corresponding fragmentation strategy to obtain a plurality of file fragments, the file fragments may be transmitted to the server 402 in a certain order.

[0180] Optionally, after the server 402 receives the multiple file segments uploaded by the client 401, it may perform corresponding decryption, decompression, etc. on the multiple file segments. Each file segment may be written to a corresponding location in the storage space in order.

[0181] In this embodiment, a file transmission system is provided. A client 401 sends a file upload request to a server 402, wherein the file upload request is used to request uploading of a target file to the server. A sharding strategy matching the target file is sent to the client by the server 402, wherein the sharding strategy indicates a rule for client 401 to perform sharding on the target file to obtain multiple file shards. The system obtains the multiple file shards uploaded by client 401, and writes the obtained file shards to storage space in server 402. This achieves the technical effect of effectively transmitting files and resolves the technical problem of being unable to effectively transmit files.

[0182] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, such as the data for verification, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0183] Example 3

[0184] Currently, when transferring files from a local client to a remote storage medium on a server, if the file to be transferred is extremely large, such as a terabyte (TB) file, it is subject to a series of limitations, such as memory on the client or server. This requires direct compression or encryption for file transfer. This process can easily strain the performance of the server or client, requiring them to achieve an extremely high level of performance advantage. This can easily lead to server or client failures, resulting in service unavailability, and other issues. Therefore, the technical problem of ineffective file transfer still exists.

[0185] Furthermore, the present application provides a method for efficient and stable transmission of very large files. If the client needs to transfer the target file to the storage space of the server, the method can trigger a corresponding file upload request and send the request to the server. After the server detects the file upload request, it needs to verify the file upload request, that is, the server determines whether the client is allowed to upload the target file. If allowed, a sharding strategy that matches the target file can be determined and sent to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transmitted according to the sharding strategy to obtain multiple file shards. The client can send the file shards to the server to write them to the storage space in the server. Since the embodiments of the present application take into account that file transfer through a single encryption or compression method will lead to performance bottleneck, a larger target file can be fragmented to obtain multiple file fragments, and each file fragment can be encrypted, compressed, etc. separately. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of performance bottleneck, thereby meeting the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transmitting files and solving the technical problem of being unable to effectively transmit files.

[0186] The above method of this embodiment is further introduced below.

[0187] In this embodiment, Figure 5 is a schematic diagram of an efficient and stable transmission architecture for large files according to an embodiment of the present application, such as Figure 5 As shown, the efficient and stable large file transmission architecture 500 may include: a user layer 501, a client 502, a service layer 503, and a storage medium 504. The user layer may include multi-system support 5011 and multi-source support 5012. The client 502 may include single file upload 5021, folder upload 5022, file download 5023, folder download 5024, upload / download pause 5025, file compression 5026, upload / download progress 5027, upload / download list 5028, task cancellation 5029, task continuation 50210, image viewing 50211, and file encryption 50212. The service layer may include tenant isolation 5031, rate control 5032, user verification 5033, permission verification 5034, adaptive fragmentation 5035, instant transmission 5036, fragmented upload 5037, breakpoint resumption 5038, file download 5039, file download continuation 50310, LZ4 compression 50311, MD5 / completion verification 50312, file streaming sha1 50313, file decryption 50314 and operation log 50315.

[0188] Alternatively, as Figure 5 As shown, the user layer supports multiple operating systems, such as Windows, Linux, and CentOS. It also supports multiple sources, such as local / removable hard drives and NAS. The browser or app used for file transfer in the user layer client allows operations such as single file upload, folder upload, file download, folder download, upload / download pause, upload / download progress, upload / download list, task cancellation (cancel upload target file), task continuation (continue upload target file), image viewing, file compression, and file encryption.

[0189] Alternatively, as Figure 5 As shown in the figure, the service layer can perform tenant (client user) isolation, rate control, user verification, permission verification, adaptive fragmentation, instant transmission, fragment upload, breakpoint resumption, file download, file continuation, fragment MD5 verification / fragment overall verification (Total Check), large file streaming SHA1, file decryption and operation logs during the self-developed transmission process.

[0190] In this embodiment, Figure 6 FIG. 1 is a flowchart of a process for generating, transmitting, and verifying a file transfer request according to an embodiment of the present application. Figure 6 As shown, this process needs to be executed between user A, client B, and server C, and can include the following steps:

[0191] Step S601: triggering a file upload operation.

[0192] In this embodiment, if the user needs to upload a target file, the user can trigger the corresponding file upload operation. For example, the user can determine the file path (filePath) of the target file to be uploaded on the client, and call the mult ipartUpload function to divide the target file under the above file path into blocks and upload it. That is, if the user needs to upload the target file, the mult ipartUpload(filePath) function can be triggered to generate a file upload request.

[0193] Step S602: In response to the file upload operation, generate and send a file transfer request.

[0194] In this embodiment, the client may generate and send a file transfer request to the server.

[0195] Optionally, the client can perform transmission initialization, that is, through beginUploadTransaction(orgld,path+filename,filesize,streaming sha-1 (less than 50G, the last one is given for files above 50G)), and based on the file upload operation, generate a file transfer request for transmitting the corresponding target file, and send it to the server. After the user performs the corresponding operation and triggers multipartUpload(filePath) to indicate the target file to be uploaded to the server, transmission initialization can be started, that is, the start upload transaction (beginUploadTransaction) is started. The identification information (orgid) of the client that needs to upload the target file can be determined first, and the file path and file name (path+filename) of the target file that the client needs to upload, as well as the file size (filesize) of the target file can be determined. For files with a size less than 50G, the calculation method of SHA-1 at this time can be determined. For files with a size greater than 50G, SHA-1 is calculated asynchronously (streaming SHA-1 (less than 50G, the last one above 50G is given)).

[0196] Step S603: Perform flow control on the client associated with the server.

[0197] Optionally, traffic can be restricted and controlled based on the org dimension. This type of flow control can limit the request flow of a specific organization (client) to ensure the reasonable allocation of resources and system stability. Through flow control, it is possible to avoid excessive requests from a certain organization causing excessive system load, or ensure that each organization can obtain fair resource allocation.

[0198] Step S604: Verify the user's login status.

[0199] In this embodiment, the server may perform a user login status check on the user of the client that issued the file upload request according to the file upload request to determine whether the user has successfully logged in.

[0200] Step S605: Verify the user's resource authority operation.

[0201] In this embodiment, the server can determine whether the user has the permission to send the file transmission based on the file upload request and the user resource permission operation verification.

[0202] Step S606: Determine whether there is a file with the same name and the same SHA1.

[0203] In this embodiment, the server can determine whether there is a file with the same name as the target file based on the file upload request, or whether there is a file with the same file content, i.e., SHA1, as the target file to be transmitted.

[0204] In this embodiment, Figure 7 is a flowchart of a process of transmitting a target file under different circumstances according to an embodiment of the present application, such as Figure 7 As shown, this process can be performed by client B, server C, and Redis D. This process can include three cases. Case 1: If a file with the same name and the same SHA1 exists, then step S701 can be executed. Case 2: If a file with the same name and the same SHA1 does not exist, then steps S702 to S720 can be executed. Case 3: If the file shard already exists, then step S721 can be executed.

[0205] Optionally, case 1 may include the following steps:

[0206] Step S701: transfer the target file in seconds.

[0207] In this embodiment, if there is a file with the same name and the same SHA1, the server can transmit the target file to the client in seconds.

[0208] Optionally, the second scenario may include the following steps:

[0209] Step S702: Generate and store a random key and upload identifier.

[0210] In this embodiment, if there is no file with the same name and the same sha1, a random key and upload ID need to be generated, and the function redis(key; uploadID) can be called to store the random key (key) and upload ID (uploadID) in redis.

[0211] Step S703: If the storage is successful, a storage success message is sent.

[0212] In this embodiment, if redis successfully stores the random key and upload identifier, it can send information indicating that the information has been successfully stored to the server.

[0213] Step S704: Obtain sharding rules.

[0214] In this embodiment, the server can call out a fragmentation rule that matches the target file that currently needs to be uploaded.

[0215] Step S705: Send the random key, upload identifier and sharding rules.

[0216] In this embodiment, the server may return the sharding rule, the random key, and the upload identifier to the corresponding client, that is, it may feed back the uploadID+random encryption key+sharding rule to the user.

[0217] Optionally, the server may determine whether a directory exists, and if not, create it, wherein the directory may be used to receive and store the file segments to be received.

[0218] Step S706: Push the file segments to be uploaded into the stack.

[0219] In this embodiment, the client can fragment the target file according to the fragmentation rules, and stack the file fragments to be uploaded.<chunkidx,retryCount> , where chunkidx can be used to represent the stack index information, that is, the stack block index, which can be used to locate the stack where the shard information is inserted. The number of retries (retryCount) can be used to indicate whether there is an exception when the shard information is pushed into the stack, and the number of times the shard information is re-pushed into the stack if an exception occurs.

[0220] Step S707: Pop the file segments from the stack.

[0221] In this embodiment, the client may pop the file segments in a corresponding order to transmit the file segments to the server.

[0222] Step S708: Calculate sha1, compress, and encrypt the file segments.

[0223] In this embodiment, when the file segments are popped out in the corresponding order, the file segment upload can be started. The client can perform sha1 calculation on the segments, IZ4 algorithm for compression, and national secret encryption.

[0224] Step S709: Send the file segments to the server.

[0225] In this embodiment, the client can send file segments to the server, that is, the client can implement the block upload operation of the target file through the following function. The function for performing the block upload operation is as follows: uploadMultPart(uploadID, fileName, fileData.fileSize, chunks, chunk, chunkSha1). During the execution of the file upload (uploadMultPart), the upload identifier (uploadID) corresponding to the target file, the file name (fileName) of the target file to be uploaded, and the file size (fileData.fileSize) of the target file can be determined. The number of segment files (chunks) into which the target file needs to be segmented and the sha1 hash value (chunkSha1) corresponding to each segment file (chunk) can also be determined. Through the above function, a larger target file can be divided into multiple segment files, and multiple segment files can be sent to the server.

[0226] Step S710: Verify the user's login status.

[0227] In this embodiment, after receiving the file segments transmitted by the client, the server may perform a user login status check.

[0228] Step S711: Verify the user's target authority.

[0229] In this embodiment, after receiving the file segments transmitted by the client, the server may further perform a user target authority check.

[0230] Step S712, determine whether the file sha1 exists.

[0231] In this embodiment, after receiving the file segment transmitted by the client, the server may confirm with Redis whether the file sha1 exists.

[0232] Step S713, return data.

[0233] In this embodiment, Redis can feedback to the server whether data corresponding to the file sha1 exists.

[0234] Step S714: decrypt the file.

[0235] In this embodiment, the server can use the corresponding key to decrypt the file.

[0236] Step S715: decompress the file.

[0237] In this embodiment, the server can decompress the file to obtain the original file segments.

[0238] Step S716: Calculate whether sha1 is consistent with the file, and report an error if not.

[0239] In this embodiment, it is possible to calculate whether sha1 is consistent with the file. If so, the other file segments can be received. If not, an error is reported.

[0240] Optionally, after decompressing the received compressed file fragments to obtain the original file fragments, the sha1 value of each decompressed original file fragment can be calculated and compared with the sha1 value of each fragment file in the client before the upload is performed to determine whether the two corresponding sha1 values ​​are consistent, so as to determine whether each file fragment is uploaded correctly. If the two sha1 values ​​are consistent, the fragment file with the consistent sha1 value can be obtained, that is, the file object with the consistent comparison can be obtained. The file pointer of the fragment file can be obtained (get file Pointer), where the file pointer can be used to indicate the location of the fragment file.

[0241] Alternatively, if the two sha1 values ​​are inconsistent, it can be determined that the file segment received by the server is different from the file segment that the client needs to upload, that is, an upload error occurs, and an error can be reported. The file segment with inconsistent sha1 values ​​can be re-uploaded.

[0242] Step S717: Calculate the fragment offset.

[0243] In this embodiment, the fragment offset can be calculated.

[0244] Optionally, since the position of each file slice in the target file is different when the target file is divided into multiple file slices for transmission, the server needs to calculate the data offset of each file slice based on the information in the request message to ensure that each file slice can be correctly written to the corresponding position of the target file. In this way, after receiving the various file slices of the target file, the server can correctly reassemble the slice data into the target file based on these data offsets.

[0245] Step S718: Write the file fragment using the offset.

[0246] In this embodiment, the file shards may be written via offsets.

[0247] Optionally, after the server obtains each file segment, it may calculate an offset position of each file segment, and write the file segment into the storage space according to the offset position.

[0248] Step S719: Check whether the file transfer is complete.

[0249] In this embodiment, Redis can be queried to see whether the file transfer is complete, that is, whether the file segments are transferred. If not, the segment sha1 is checked and success is returned.

[0250] Step S720, return the result.

[0251] In this embodiment, Redis can feedback the result of whether the transmission is completed to the server.

[0252] Optionally, case three may include the following steps:

[0253] Step S721: Return a message indicating that the file segment transmission is successful.

[0254] In this embodiment, if the file segment transmission is completed, the server can feed back a message of successful transmission to the client.

[0255] In this embodiment, Figure 8 This is a flowchart of an execution process when file segmentation is not completed and file transfer is completed according to an embodiment of the present application. Figure 8 As shown, this process can be performed by client B, server C, Redis D, and relational database service (Rds) E. This process can include two cases. Case 1: if file sharding is not completed, then steps S801 to S804 can be executed. Case 2: if file transfer is completed, then steps S805 to S812 can be executed.

[0256] Optionally, scenario 1 may include the following steps:

[0257] Step S801: Determine the MD5 value of each file segment to verify the integrity of the file.

[0258] In this embodiment, the fragment MD5 may be checked.

[0259] Step S802: Successfully write the file segment + update.

[0260] In this embodiment, the server can write file shards to redis and can update the original file shards in redis.

[0261] Step S803: Return the result indicating successful writing.

[0262] In this embodiment, if redis successfully writes the file segment, it can feed back information indicating the writing success to the server.

[0263] Step S804: Return the result of successful transmission.

[0264] In this embodiment, if the file segment is written successfully, the server can feed back information indicating the writing success to the client.

[0265] Optionally, the second scenario may include the following steps:

[0266] Step S805: Calculate the checksum of the file using a hash algorithm.

[0267] In this embodiment, a hash algorithm may be used to determine sha1 and calculate the checksum of the file, that is, perform checkSum Sha1 file verification.

[0268] Step S806: rename the file.

[0269] In this embodiment, the temp file can be renamed and a send event (send envent) can be triggered. The renamed file can be uploaded to Rds through send envent (rocketmq).

[0270] Step S807: write file information.

[0271] In this embodiment, file information (space, information) can be written to Rds in an asynchronous process.

[0272] Step S808: If the file information is written successfully, a successful writing information is returned to prompt the server.

[0273] In this embodiment, if the writing is successful, Rds can feed back the writing success information to the server.

[0274] Step S809: writing the operation log.

[0275] In this embodiment, an operation log may be written to Rds.

[0276] Optionally, after the acquired file segments are written to the storage space in the server, an operation log of the target file may be recorded.

[0277] Optionally, after the transfer of each file segment corresponding to the target file is completed, the operation flow can be recorded.

[0278] Step S810: If the operation log is written successfully, a successful writing message is returned to prompt the server.

[0279] In this embodiment, if the operation log is successfully written, Rds can feed back corresponding prompt information to the server.

[0280] Step S811, clearing the information corresponding to the upload identifier.

[0281] In this embodiment, the information corresponding to the uploadID in redis may be cleared.

[0282] Step S812: Return the result of successful cleaning.

[0283] In this embodiment, if the cleanup is successful, redis can feed back corresponding information to the server as a prompt.

[0284] In this embodiment, Figure 9 This is a flowchart of a solution to a failure in uploading a file in segments according to an embodiment of the present application. Figure 9 As shown in the figure, the process can be performed by client B, server C and Redis D. The process can be performed as follows:

[0285] Step S901: Send a message indicating that the file segment upload has failed.

[0286] In this embodiment, the situation that file segment upload fails can include server error and network transmission error.If upload fails, then the server can send a message (failure) of upload failure to the client.

[0287] Step S902: Determine whether to proceed with the upload failure process.

[0288] In this embodiment, it can be determined whether it is necessary to go through the process of re-uploading if the upload fails.

[0289] Optionally, by determining the size relationship between retCount and maxRetryCount, if retCount is greater than maxRetryCount, the upload failure process needs to be performed.

[0290] Step S903: Create a stack for the file segments that need to be re-uploaded.

[0291] In this embodiment, a new stack can be created for the file segments that need to be re-uploaded. The file segments that failed to be uploaded can be re-stacked into the stack (pushStack), and the index information (chunkidx) of the file segments that failed to be uploaded in the stack can be recorded. Since this is a re-upload process after confirming that the file segment has failed to be uploaded, the number of retries at this time needs to be increased by one (retryCount+1), that is, else: pushStack(chunkidx, retryCount+1) to pop the file segment out of the stack and transmit it to the server.

[0292] Step S904: Release the lock.

[0293] In this embodiment, the lock of the concurrent thread of the upload task in redis can be released, that is, the lock is released, and the related resources in the transmission process can be released.

[0294] Step S905, return message.

[0295] In this embodiment, Redis may feed back to the server a message indicating whether the release is successful.

[0296] Step S906: Send the successfully uploaded file segments.

[0297] In this embodiment, if the release is successful, the server may send the successfully uploaded file segments to the client to prompt the client that the file segments that failed to be uploaded have been successfully re-uploaded.

[0298] Step S907: record success information.

[0299] In this embodiment, the client can record the information of the successful re-upload: uploadRecord.done.push(chunkidx), uploadRecord.needUload.remove(chunkidx), where uploadRecord can be used to represent the upload record of the operation of uploading the segmented file to the server. uploadRecord.needUpload.remove(chunkidx) can be used to indicate the removal of the chunkidx corresponding to the successfully uploaded file segment from the stack that needs to be uploaded to the server. uploadRecord.done.push(chunkidx) indicates that the chunkidx corresponding to the successfully uploaded file segment is added to the record of the completed upload.

[0300] Step S908: If the number of times exceeds the retry threshold, the file upload operation is terminated.

[0301] In this embodiment, after entering the upload failure process, the relationship between the number of retries and the retry threshold can be determined. If the number of retries is greater than the retry threshold, the file upload operation can be terminated. If the number of retries is less than or equal to the retry threshold, the file segment can be re-uploaded.

[0302] Step S909: Clear the local file upload information and end the upload.

[0303] In this embodiment, if it is necessary to terminate the file upload operation, the local upload record (uploadRecord) can be cleared to terminate the upload.

[0304] For example, Figure 10 is a schematic diagram of an end-to-end interaction for efficient and stable transmission of ultra-large files according to an embodiment of the present application. Figure 10 As shown, the interaction process may include a browser 101, a client 102, and a service (Server) 103. The browser 101 may include file management 1011, space management 1012, and a recycle bin 1013. The client 102 may include a browser project 1021 and NodeJS 1022. The browser project 1021 may include account login 10211, file upload 10212, and file download 10213. NodeJS 1022 may include a router 10221 and a DataQ executable file 10222, and may also include file encryption, MD5 calculation, file sharding, token management, file compression, and monitoring tracking. The service 103 may include a data quality service (DataQ Server) 1031 and an intelligent computing service 1032. The browser and client may interact using schema:icapp: / / xxx. The client and service may interact using http. Inter-process communication is performed between the browser project and NodeJS. The Router and DataQ Server communicate with each other through inter-process communication, where the token can be used to represent user identity, permissions, or access credentials.

[0305] Alternatively, a user initiates a file upload request through an application (app). The request reaches the server via an HTTP stream, where the server logs in and verifies resource permissions. After the permissions are verified, a name and SHA1 check is performed. If the file exists, an idempotent response is returned. Otherwise, a normal file upload proceeds to the next step, where the server generates a random key and uploadId and returns the relevant information to the app.

[0306] Optionally, after obtaining the encryption key and uploadId, the app performs file encryption, sharding, compression, and other operations. The app uploads the processed file shards to the server. After receiving the request, the server takes the key from Redis according to the uploadId to decrypt the shard file. After successful decryption, it decompresses the file and obtains the file stream.

[0307] Optionally, based on the request message, the offset position of the file fragment stream data is calculated, written to the disk through nio memory mapping, and the location is recorded. If all locations are transferred, the sha1 value of the file is calculated to ensure the consistency of the file.

[0308] Optionally, after the file transfer is completed, the operation flow will be recorded, the concurrent lock will be released, and related resources will be released.

[0309] In an embodiment of the present application, if the client needs to transfer a target file to the storage space of the server, a corresponding file upload request can be triggered and the request can be sent to the server. After the server detects the file upload request, it can determine a sharding strategy that matches the target file and send it to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transferred according to the sharding strategy to obtain multiple file shards. The client can send the file shards to the server to write them to the storage space in the server. Since the embodiment of the present application takes into account that if a file is transferred by a single encryption or compression method, it will lead to a situation where performance is fully utilized, a larger target file can be sharded to obtain multiple file shards, and each file shard can be encrypted, compressed, etc. respectively. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of full performance, thereby achieving the ability to meet the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transferring files and solving the technical problem of being unable to effectively transfer files.

[0310] Example 4

[0311] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 2 The file transmission method shown is a file transmission device.

[0312] Figure 11 is a schematic diagram of a file transmission device according to an embodiment of the present application, such as Figure 11 As shown, the file transmission device 1100 may include: a first sending unit 1102 , an acquiring unit 1104 and a writing unit 1106 .

[0313] The first sending unit 1102 is configured to send a fragmentation strategy that matches the target file to the client, wherein the fragmentation strategy is used to indicate a rule for the client to fragment the target file to obtain multiple file fragments.

[0314] The acquiring unit 1104 is configured to acquire multiple file segments uploaded by the client.

[0315] The writing unit 1106 is used to write the acquired file segments into the storage space in the server.

[0316] Here, the acquisition unit, verification unit, first sending unit 1102, acquisition unit 1104, and writing unit 1106 correspond to steps S202 to S206 in Example 1. The examples and application scenarios implemented by the three units and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 1304) and processed by one or more processors (e.g., processors 1302a, 1302b..., 1302n). The above-mentioned units can also be part of the device and can be run in the computer terminal A provided in Example 5.

[0317] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 4 The file transmission method shown is a file transmission device.

[0318] Figure 12 is a schematic diagram of a file transmission device according to an embodiment of the present application, such as Figure 12 As shown, the file transmission device 1200 may include: a second sending unit 1202 , a receiving unit 1204 , a fragmentation unit 1206 and an uploading unit 1208 .

[0319] The second sending unit 1202 is configured to send a file upload request to the server, wherein the file upload request is used to request uploading a target file to the server.

[0320] The receiving unit 1204 is configured to receive a fragmentation strategy that matches the target file and is sent by the server in response to the file upload request, wherein the fragmentation strategy is used to represent a rule for fragmenting the target file.

[0321] The fragmentation unit 1206 is used to fragment the target file according to the fragmentation strategy to obtain multiple file fragments.

[0322] The uploading unit 1208 is configured to upload multiple file segments to the server, wherein the file segments are stored in a storage space by the server.

[0323] It should be noted that the second sending unit 1202, receiving unit 1204, sharding unit 1206, and uploading unit 1208 correspond to steps S302 to S308 in Example 1. The four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 1304) and processed by one or more processors (e.g., processors 1302a, 1302b..., 1302n). The above-mentioned units can also be part of the device and can be run in the computer terminal A provided in Example 5.

[0324] In the file transmission device, if the client has a need to transfer the target file to the storage space of the server, a corresponding file upload request can be triggered and the request can be sent to the server. After the server detects the file upload request, it can determine the sharding strategy that matches the target file and send it to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transferred according to the sharding strategy to obtain multiple file shards. The client can send the file shards to the server to write them to the storage space in the server. Since the embodiment of the present application takes into account that if a file is transferred by a single encryption or compression method, it will lead to a situation where the performance is full, it can be processed by sharding a larger target file to obtain multiple file shards, and each file shard can be encrypted, compressed, etc. respectively. This can greatly solve the memory usage problem in the client and server, and can also avoid the problem of full performance, thereby achieving the ability to meet the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transmitting files and solving the technical problem of being unable to effectively transmit files.

[0325] Example 5

[0326] The embodiment of the present application can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0327] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0328] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the file transmission method: in response to a file upload request from the client, a sharding strategy matching the target file is sent to the client, wherein the file upload request is used to request uploading the target file to the server, and the sharding strategy is used to indicate that the client performs sharding processing on the target file to obtain a rule for multiple file shards; obtaining multiple file shards uploaded by the client; and writing the obtained file shards to the storage space in the server.

[0329] Optionally, Figure 13 This is a structural block diagram of a computer terminal according to an embodiment of the present application. Figure 13 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1302 , a memory 1304 and a transmission device 1306 .

[0330] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the file transmission method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned file transmission method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to terminal A via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0331] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: if the file attribute of the target file is greater than the file attribute threshold, obtain a sharding strategy that matches the file attribute of the target file, wherein the file attribute is used to represent the size of the target file.

[0332] Optionally, the processor may further execute program code of the following steps: determining the operating system on which the client is running; if the file attribute of the target file is greater than a file attribute threshold, obtaining a sharding strategy that matches both the operating system and the file attribute.

[0333] Optionally, the above-mentioned processor can also execute the program code of the following steps: obtain the encryption key and upload identifier corresponding to the target file, wherein the encryption key is used to encrypt the target file, and the upload identifier is used to identify the upload process of the target file; store the encryption key and upload identifier; and obtain the sharding strategy when the encryption key and upload identifier are successfully stored.

[0334] Optionally, the processor may also execute the following program code: when the encryption key and upload identifier are successfully stored, the encryption key and upload identifier are sent to the client; multiple file segments uploaded by the client are obtained, including: obtaining multiple encrypted file segments uploaded by the client according to the upload identifier, wherein the encrypted file segments are obtained by the client encrypting the file segments according to the encryption key; calling the stored encryption key according to the upload identifier, decrypting the encrypted file segments, and obtaining the file segments.

[0335] Optionally, the processor may also execute the following program code: the encrypted file segments are compressed by the client, and the compression results are encrypted according to the encryption key, wherein the stored encryption key is called according to the upload identifier, and the encrypted file segments are decrypted to obtain the file segments, including: calling the stored encryption key according to the upload identifier, decrypting the encrypted file segments to obtain the compression results; and decompressing the compression results to obtain the file segments.

[0336] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the data offset of the obtained file segment in the target file, wherein the data offset is used to indicate the position of the file segment in the target file; based on the data offset, write the file segment into the storage space.

[0337] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the file content of the file segment to be mapped to the write position in the storage space according to the data offset; and write the file content of the file segment to the write position.

[0338] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the first hash value of the obtained file segment on the server, wherein the first hash value is used to represent the file content of the file segment on the server; if the first hash value is the same as the second hash value of the file segment on the client, write the obtained file segment to the storage space in the server, wherein the second hash value is used to represent the file content of the file segment on the client.

[0339] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: respond to the file upload request from the client, verify the file upload request, and obtain the verification result; if the verification result is that the server allows the client to upload the target file, then send the sharding strategy that matches the target file to the client.

[0340] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to a file upload request from the client, perform at least one of the following verifications on the file upload request to obtain a verification result; verify the login status of the client's account in the file upload request; verify the resource operation permissions associated with the client's account in the file upload request, wherein the resource operation permissions are used to limit the client's flow; verify the target identification information of the target file in the file upload request, wherein the target identification information is used to represent the file name and file content of the target file.

[0341] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to the login status verification of the account passing, the resource operation permission verification passing, and the absence of identification information identical to the target identification information in the server, determine that the verification result is that the server allows the client to transfer the target file.

[0342] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: adjust the sharding strategy based on at least one of the following information: the performance indicators of the server, the performance indicators of the client, and the network status between the server and the client.

[0343] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to the failure to obtain the file fragment uploaded by the client, send a retransmission request to the client, wherein the retransmission request is used to request the client to upload the file fragment to the server again.

[0344] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: record the operation log corresponding to the target file, wherein the operation log is used to indicate the operation process of uploading the target file from the client to the server.

[0345] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: sending a file upload request to the server, wherein the file upload request is used to request uploading the target file to the server; receiving a sharding strategy that matches the target file and is sent by the server in response to the file upload request, wherein the sharding strategy is used to represent the rules for sharding the target file; sharding the target file according to the sharding strategy to obtain multiple file shards; uploading the multiple file shards to the server, wherein the file shards are stored in the storage space by the server.

[0346] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: compress the file fragments to obtain compression results; encrypt the compression results according to the encryption key from the server to obtain encrypted file fragments.

[0347] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: upload the encrypted file segments to the server according to the upload identifier from the server, wherein the upload identifier is used to identify the upload process of the target file, and the encrypted file segments are called by the server according to the upload identifier. The stored encryption key is used to decrypt the encrypted file segments to obtain the compression result, and the compression result is decompressed by the server to obtain the file segments.

[0348] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: sending a file download request to the server, wherein the file download request is used to request the server to transmit the target file to the client; obtaining multiple file fragments sent by the server in response to the file download request; and generating a target file from the multiple file fragments.

[0349] An embodiment of the present application provides a file transfer method. In this embodiment, if a client needs to transfer a target file to a server's storage space, it can trigger a corresponding file upload request and send the request to the server. After the server detects the file upload request, it can determine a sharding strategy that matches the target file and send it to the client. After receiving the sharding strategy, the client can perform corresponding sharding processing on the target file to be transferred according to the sharding strategy to obtain multiple file shards. The client can send the file shards to the server to be written to the server's storage space. Since this embodiment of the present application takes into account that file transfer using a single encryption or compression method will result in performance saturation, it can be achieved by sharding a larger target file to obtain multiple file shards. Each file shard can be encrypted, compressed, etc. separately. This can significantly solve the memory usage problem in the client and server, and can also avoid performance saturation, thereby meeting the needs of file transfer. For example, for the special needs of ultra-large file transfer, the efficiency and security of file transfer are improved, thereby achieving the technical effect of effectively transferring files and solving the technical problem of being unable to effectively transfer files.

[0350] It can be understood by those skilled in the art that Figure 13 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, abbreviated as MID), a PAD, or other terminal devices. Figure 13 It does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include Figure 13 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 13 Different configurations shown.

[0351] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0352] Example 6

[0353] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the file transmission method provided in the first embodiment.

[0354] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0355] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to a file upload request from the client, sending a sharding strategy that matches the target file to the client, wherein the file upload request is used to request uploading the target file to the server, and the sharding strategy is used to indicate that the client performs sharding processing on the target file to obtain a rule for multiple file shards; obtaining multiple file shards uploaded by the client; and writing the obtained file shards to the storage space in the server.

[0356] Optionally, the computer-readable storage medium may further execute program code of the following steps: if the file attribute of the target file is greater than a file attribute threshold, obtaining a sharding strategy that matches the file attribute of the target file, wherein the file attribute is used to indicate the size of the target file.

[0357] Optionally, the computer-readable storage medium may further execute program code for the following steps: determining the operating system on which the client is running; if the file attribute of the target file is greater than a file attribute threshold, obtaining a sharding strategy that matches both the operating system and the file attribute.

[0358] Optionally, the above-mentioned computer-readable storage medium can also execute the program code of the following steps: obtaining the encryption key and upload identifier corresponding to the target file, wherein the encryption key is used to encrypt the target file, and the upload identifier is used to identify the upload process of the target file; storing the encryption key and upload identifier; and obtaining the sharding strategy when the encryption key and upload identifier are successfully stored.

[0359] Optionally, the computer-readable storage medium can also execute the program code of the following steps: when the encryption key and upload identifier are successfully stored, the encryption key and upload identifier are sent to the client; multiple file fragments uploaded by the client are obtained, including: obtaining multiple encrypted file fragments uploaded by the client according to the upload identifier, wherein the encrypted file fragments are obtained by the client encrypting the file fragments according to the encryption key; calling the stored encryption key according to the upload identifier, decrypting the encrypted file fragments, and obtaining the file fragments.

[0360] Optionally, the computer-readable storage medium can also execute the following program code: the encrypted file segments are compressed by the client, and the compression results are encrypted according to the encryption key, wherein the stored encryption key is called according to the upload identifier, and the encrypted file segments are decrypted to obtain the file segments, including: calling the stored encryption key according to the upload identifier, decrypting the encrypted file segments to obtain the compression results; decompressing the compression results to obtain the file segments.

[0361] Optionally, the above-mentioned computer-readable storage medium can also execute the program code of the following steps: determining the data offset of the acquired file fragment in the target file, wherein the data offset is used to indicate the position of the file fragment in the target file; based on the data offset, writing the file fragment into the storage space.

[0362] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: determining, according to a data offset, mapping the file content of a file segment to a write position in a storage space; and writing the file content of the file segment to the write position.

[0363] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: determining a first hash value of the obtained file shard on the server, wherein the first hash value is used to represent the file content of the file shard on the server; if the first hash value is the same as a second hash value of the file shard on the client, writing the obtained file shard to a storage space in the server, wherein the second hash value is used to represent the file content of the file shard on the client.

[0364] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: in response to a file upload request from a client, verifying the file upload request to obtain a verification result; if the verification result is that the server allows the client to upload the target file, sending a sharding strategy that matches the target file to the client.

[0365] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: in response to a file upload request from a client, performing at least one of the following checks on the file upload request to obtain a verification result; verifying the login status of the client's account in the file upload request; verifying the resource operation permissions associated with the client's account in the file upload request, wherein the resource operation permissions are used to limit the client's flow; verifying the target identification information of the target file in the file upload request, wherein the target identification information is used to represent the file name and file content of the target file.

[0366] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: in response to the account login status verification being passed, the resource operation permission verification being passed, and the absence of identification information identical to the target identification information in the server, determining that the verification result is that the server allows the client to transfer the target file.

[0367] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: adjusting the sharding strategy based on at least one of the following information: performance indicators of the server, performance indicators of the client, and the network status between the server and the client.

[0368] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: in response to a failure to obtain a file segment uploaded by a client, sending a retransmission request to the client, wherein the retransmission request is used to request the client to upload the file segment to the server again.

[0369] As an optional example, the computer-readable storage medium is configured to store program code for executing the following steps: recording an operation log corresponding to the target file, wherein the operation log is used to indicate the operation process of uploading the target file from the client to the server.

[0370] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: sending a file upload request to a server, wherein the file upload request is used to request uploading a target file to the server; receiving a sharding strategy that matches the target file and is sent by the server in response to the file upload request, wherein the sharding strategy is used to represent rules for sharding the target file; sharding the target file according to the sharding strategy to obtain multiple file shards; uploading the multiple file shards to the server, wherein the file shards are stored in a storage space by the server.

[0371] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: compressing the file segments to obtain compression results; encrypting the compression results according to an encryption key from a server to obtain encrypted file segments.

[0372] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: uploading the encrypted file segments to the server according to an upload identifier from the server, wherein the upload identifier is used to identify the upload process of the target file, the encrypted file segments are called by the server according to the upload identifier using a stored encryption key, the encrypted file segments are decrypted to obtain a compression result, and the compression result is decompressed by the server to obtain the file segments.

[0373] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: sending a file download request to a server, wherein the file download request is used to request the server to transfer a target file to a client; obtaining multiple file fragments sent by the server in response to the file download request; and generating a target file from the multiple file fragments.

[0374] Example 7

[0375] An embodiment of the present application may provide an electronic device, which may include a memory and a processor.

[0376] Figure 14 1 is a block diagram of an electronic device for a file transmission method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0377] like Figure 14As shown, device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1402 or a computer program loaded from a storage unit 1408 into a random access memory (RAM) 1403. Various programs and data required for the operation of device 1400 can also be stored in RAM 1403. Computing unit 1401, ROM 1402, and RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to bus 1404.

[0378] Various components in device 1400 are connected to I / O interface 1405, including: an input unit 1406, such as a keyboard, mouse, etc.; an output unit 1404, such as various types of displays, speakers, etc.; a storage unit 1408, such as a magnetic disk, optical disk, etc.; and a communication unit 1409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0379] The computing unit 1401 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as the file transmission method. For example, in some embodiments, the file transmission method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the file transmission method described above may be performed. Alternatively, in other embodiments, the computing unit 1401 may be configured to execute the file transmission method in any other appropriate manner (e.g., by means of firmware).

[0380] Example 8

[0381] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, which, when executed by a processor, implements the file transmission method of the embodiment of the present application.

[0382] According to an embodiment of the present application, a file transmission method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0383] The method embodiment provided in Example 8 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 15 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a file transmission method according to an embodiment of the present application, such as Figure 15 As shown, the computer terminal 150 (or mobile device) may include one or more (illustrated by 1502a, 1502b, ..., 1502n in the figure) processors 1502 (the processor 1502 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, abbreviated as MCU) or a programmable logic device (Field Programmable Gate Array, abbreviated as FPGA)), a memory 1504 for storing data, and a transmission device 1506 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (Universal Serial Bus, abbreviated as USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 15 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 15 More or fewer components than shown, or with Figure 15 Different configurations shown.

[0384] Figure 15 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the computer terminal 150 (or mobile device), but also as an exemplary block diagram of the server. In an optional embodiment, Figure 16 The block diagram shows the use of the above Figure 15 The computer terminal 150 (or mobile device) is shown as an embodiment of a computing node in the computing environment 1601.

[0385] Figure 16 This is a structural block diagram of a computing environment of a file transmission method according to an embodiment of the present application, such as Figure 16 As shown, computing environment 1601 includes multiple computing nodes (e.g., servers) (illustrated as 1610-1, 1610-2, ...) running on a distributed network. Each computing node includes local processing and memory resources, and end users 1602 can remotely run applications or store data in computing environment 1601. Applications can be provided as multiple services 1620-1, 1620-2, 1620-3, and 1620-4 in computing environment 1601, representing services "F," "G," "I," and "H," respectively.

[0386] End user 1602 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or request of end user 1602 can be provided to the entry gateway 1630. The entry gateway 1630 can include a corresponding agent to handle the provisioning and / or request for services (one or more services provided in the computing environment 1601).

[0387] Services are provided or deployed based on various virtualization technologies supported by the computing environment 1601. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly contacting any actual hardware resources. While virtual machines virtualize machines, container-based virtualization can start containers to virtualize the entire operating system so that multiple workloads can run on a single operating system instance.

[0388] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 16 As shown, service 1620-2 can be configured with one or more Pods 1640-1, 1640-2, ..., 1640-N (collectively, Pods). A Pod can include a proxy 1645 and one or more containers 1642-1, 1642-2, ..., 1642-M (collectively, containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. Proxy 1645 typically controls network functions related to the service, such as routing and load balancing. Other services can also be configured with Pods similar to Pods.

[0389] During operation, executing a user request from the end user 1602 may require calling one or more services in the computing environment 1601, and executing one or more functions of a service may require calling one or more functions of another service. Figure 16 As shown, service "F" 1620-1 receives a user request from end user 1602 from ingress gateway 1630, service "F" 1620-1 may call service "G" 1620-2, and service "G" 1620-2 may request service "I" 1620-3 to perform one or more functions.

[0390] This computing environment can be a cloud computing environment, where resource allocation is managed by the cloud service provider, allowing for feature development without having to worry about implementing, adjusting, or scaling servers. This computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be partitioned to perform a set of functions that can scale independently and automatically, rather than scaling a single hardware device to handle the potential load.

[0391] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0392] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0393] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0394] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0395] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0396] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD)) for displaying information to the user, a monitor; and a keyboard and pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0397] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0398] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0399] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0400] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0401] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0402] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0403] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0404] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk.

[0405] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A file transmission method, characterized in that: Applicable to servers, including: In response to a file upload request from a client, sending a fragmentation strategy matching a target file to the client, wherein the file upload request is used to request uploading the target file to the server, and the fragmentation strategy is used to indicate a rule for the client to fragment the target file to obtain multiple file fragments; Obtaining the multiple file fragments uploaded by the client; The obtained file segments are written into the storage space in the server.

2. The method according to claim 1, characterized in that The method further comprises: If the file attribute of the target file is greater than the file attribute threshold, the sharding strategy matching the file attribute of the target file is obtained, wherein the file attribute is used to indicate the size of the target file.

3. The method according to claim 2, characterized in that If the file attribute of the target file is greater than the file attribute threshold, obtaining the fragmentation strategy that matches the file attribute of the target file includes: Determining the operating system on which the client is running; If the file attribute of the target file is greater than the file attribute threshold, the sharding strategy that matches both the operating system and the file attribute is obtained.

4. The method according to claim 1, wherein The method further comprises: Obtaining an encryption key and an upload identifier corresponding to the target file, wherein the encryption key is used to encrypt the target file, and the upload identifier is used to identify the upload process of the target file; Storing the encryption key and the upload identifier; When the encryption key and the upload identifier are successfully stored, the sharding strategy is obtained.

5. The method according to claim 4, characterized in that The method further comprises: If the encryption key and the upload identifier are successfully stored, sending the encryption key and the upload identifier to the client; Obtaining the multiple file fragments uploaded by the client includes: obtaining the multiple encrypted file fragments uploaded by the client according to the upload identifier, wherein the encrypted file fragments are obtained by the client encrypting the file fragments according to the encryption key; calling the stored encryption key according to the upload identifier, decrypting the encrypted file fragments, and obtaining the file fragments.

6. The method according to claim 1, wherein Writing the obtained file segments into the storage space in the server includes: Determine a data offset of the obtained file fragment in the target file, wherein the data offset is used to indicate a position of the file fragment in the target file; Based on the data offset, the file segment is written into the storage space.

7. The method according to claim 6, characterized in that Writing the file fragment into the storage space based on the data offset includes: Determining, according to the data offset, mapping the file content of the file slice to a write position in the storage space; Write the file content of the file segment to the write location.

8. The method according to claim 1, characterized in that Writing the obtained file segments into the storage space in the server includes: Determine a first hash value of the obtained file segment on the server, wherein the first hash value is used to represent the file content of the file segment on the server; If the first hash value is the same as the second hash value of the file segment on the client, the acquired file segment is written to the storage space in the server, wherein the second hash value is used to represent the file content of the file segment on the client.

9. The method according to claim 1, characterized in that In response to a file upload request from a client, a sharding strategy matching the target file is sent to the client, including: In response to the file upload request from the client, verify the file upload request and obtain a verification result; If the verification result is that the server allows the client to upload the target file, the sharding strategy matching the target file is sent to the client.

10. The method according to claim 9, characterized in that In response to a file upload request from a client, verify the file upload request to obtain a verification result, including: In response to the file upload request from the client, perform at least one of the following verifications on the file upload request to obtain the verification result: Verify the login status of the client account in the file upload request: Verifying the resource operation permission associated with the client's account in the file upload request, wherein the resource operation permission is used to limit the client's access; The target identification information of the target file in the file upload request is verified, wherein the target identification information is used to represent the file name and file content of the target file.

11. The method according to claim 10, characterized in that The method further comprises: In response to the account login status verification being passed, the resource operation permission verification being passed, and the server not having identification information identical to the target identification information, it is determined that the verification result is that the server allows the client to transfer the target file.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: The sharding strategy is adjusted based on at least one of the following information: a performance indicator of the server, a performance indicator of the client, and a network status between the server and the client.

13. A file transmission method, characterized in that: Applied to the client, including: Sending a file upload request to a server, wherein the file upload request is used to request uploading a target file to the server; receiving a fragmentation strategy matching the target file and sent by the server in response to the file upload request, wherein the fragmentation strategy is used to represent a rule for fragmenting the target file; Slice the target file according to the slicing strategy to obtain multiple file slicings; The plurality of file segments are uploaded to the server, wherein the file segments are stored in a storage space by the server.

14. The method according to claim 13, characterized in that The method further comprises: Compressing the file segments to obtain compression results; The compression result is encrypted according to the encryption key from the server to obtain the encrypted file fragments.

15. The method according to claim 14, characterized in that Uploading the plurality of file segments to the server includes: According to the upload identifier from the server, the encrypted file segments are uploaded to the server, wherein the upload identifier is used to identify the upload process of the target file. The encrypted file segments are called by the server according to the upload identifier to store the encryption key, and the encrypted file segments are decrypted to obtain the compression result. The compression result is decompressed by the server to obtain the file segments.

16. The method according to claim 13, characterized in that After uploading the plurality of file segments to the server, the method further includes: Sending a file download request to the server, wherein the file download request is used to request the server to transmit a target file to the client; Obtaining the plurality of file fragments sent by the server in response to the file download request; The target file is generated by dividing the plurality of files into segments.

17. A file transmission system, characterized in that: include: Client and server, where The client is used to send a file upload request to the server, wherein the file upload request is used to request uploading a target file to the server; The server is used to send a sharding strategy that matches the target file to the client in response to the file upload request, wherein the sharding strategy is used to indicate the rule for the client to perform sharding processing on the target file to obtain multiple file shards; obtain the multiple file shards uploaded by the client; and write the obtained file shards into the storage space in the server.

18. An electronic device, characterized in that: include: a memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 16.

20. A computer program product, characterized in that The method comprises computer instructions, which implement the method according to any one of claims 1 to 16 when executed by a processor.

Images