File transmission and storage method and device, computer device, readable storage medium and program product

Abstract

The application relates to a file transmission and storage method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: a client determines at least one of the type and size of a to-be-uploaded file by responding to a file uploading instruction of a target user, determines an uploading mode of the to-be-uploaded file based on an uploading strategy of the target user and at least one of the type and size of the to-be-uploaded file, and then stores the to-be-uploaded file to a server based on the uploading mode. The efficiency of uploading small files to a cloud disk can be improved, the pressure on underlying storage can be reduced, and the efficiency of uploading files to the cloud disk and the pressure on underlying storage can be balanced.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a file transfer and storage method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] In modern internet technology, cloud storage, as an important data storage service, has received widespread attention and application. Cloud storage allows users to store, manage, and share files on remote servers, greatly facilitating their digital lives and work. However, with the increasing demand for cloud storage, especially for storing large numbers of small files, cloud storage faces some technical challenges in file transfer and storage.

[0003] In traditional technologies, cloud storage services typically use the default file upload and storage methods. During the file upload process, if the file already exists on the server, bandwidth and transmission time can be saved through instant upload (i.e., without re-uploading the existing file).

[0004] However, traditional file upload and storage methods still have some problems when handling a large number of small files. For example, during a fast upload, it is necessary to calculate the file hash value to compare with the server data. For small files, this may result in low transmission efficiency, or even be less efficient than direct upload. Summary of the Invention

[0005] Therefore, it is necessary to provide a file transfer and storage method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve transmission efficiency in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides a file transfer and storage method, the method being applied to a client, the method comprising:

[0007] In response to a file upload instruction from a target user, at least one of the type and size of the file to be uploaded is determined, the file to be uploaded being carried in the file upload instruction;

[0008] Based on the target user's upload strategy and at least one of the type and size of the file to be uploaded, the upload method of the file to be uploaded is determined, and the upload method includes any one of instant upload, direct upload and small file upload.

[0009] The file to be uploaded is stored on the server based on the aforementioned upload method.

[0010] In one embodiment, prior to responding to the file upload instruction from the target user, the method further includes: in response to the login instruction from the target user, obtaining the login information of the target user, the login information including the user type and login device information of the target user; reporting the user type and login device information of the target user to the server to instruct the server to configure an upload strategy for the target user based on the user type and login device information of the target user; and receiving the upload strategy of the target user returned by the server.

[0011] In one embodiment, determining the upload method of the file to be uploaded based on the target user's upload strategy and at least one of the type and size of the file to be uploaded includes: if the target user's upload strategy specifies a first file size threshold for small file uploads, and the size of the file to be uploaded is determined to be less than or equal to the first file size threshold specified by the upload strategy, the upload method of the file to be uploaded is determined to be small file upload; if the target user's upload strategy specifies a first file type for instant uploads, and the type of the file to be uploaded matches the first file type specified by the upload strategy, the upload method of the file to be uploaded is determined to be instant upload; if the target user's upload strategy specifies a second file type for direct uploads and a second file size threshold, and the type of the file to be uploaded matches the second file type specified by the upload strategy, and the size of the file to be uploaded is greater than the second file size threshold, the upload method of the file to be uploaded is determined to be direct upload, where the second file size threshold is greater than the first file size threshold.

[0012] In one embodiment, storing the file to be uploaded to the server based on the upload method includes: if it is determined that the upload method of the file to be uploaded is a small file upload, sending the file content and file path of the file to be uploaded to the server; instructing the server to search for a matching file in the object storage based on the file content of the file to be uploaded; if a matching file is found, creating a business index of the file to be uploaded and returning a successful upload result; if no matching file is found, uploading the file to be uploaded to an aggregated object file based on the file path, creating a business index including the aggregated object file pointed to by the file to be uploaded, and returning a successful upload result; and receiving the successful upload result returned by the server.

[0013] In one embodiment, storing the file to be uploaded to the server based on the upload method includes: if it is determined that the upload method of the file to be uploaded is direct upload, requesting upload resources from the server based on the size of the file to be uploaded; and if the upload resources are received from the server, slicing the file to be uploaded according to the upload resources and uploading them concurrently to the server.

[0014] In one embodiment, storing the file to be uploaded to the server based on the upload method includes: if the upload method of the file to be uploaded is determined to be instant upload, determining the corresponding hash value based on the file content of the file to be uploaded; sending the hash value, file path, and file size of the file to be uploaded to the server; instructing the server to search for a matching file in object storage based on the hash value and file size of the file to be uploaded; if a matching file is found, creating a business index for the file to be uploaded and returning a result of successful instant upload; if no matching file is found, returning the upload resource; receiving the result of successful instant upload returned by the server; or, if the upload resource is received from the server, slicing the file to be uploaded according to the upload resource and uploading it concurrently to the server.

[0015] Secondly, this application provides a file transfer and storage device, which is applied to a client and includes:

[0016] The file determination module is used to determine at least one of the type and size of the file to be uploaded in response to the file upload instruction of the target user, wherein the file to be uploaded is carried in the file upload instruction;

[0017] The upload method determination module is used to determine the upload method of the file to be uploaded based on the target user's upload strategy and at least one of the type and size of the file to be uploaded. The upload method includes any one of instant upload, direct upload, and small file upload.

[0018] A storage module is used to store the file to be uploaded to the server based on the upload method.

[0019] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0021] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0022] The aforementioned file transfer and storage methods, apparatus, computer equipment, computer-readable storage media, and computer program products involve a client responding to a target user's file upload instruction, determining at least one of the type and size of the file to be uploaded, and based on the target user's upload strategy and at least one of the type and size of the file to be uploaded, determining the upload method for the file to be uploaded, and then storing the file to be uploaded to the server based on the upload method. This not only improves the efficiency of uploading small files to the cloud drive but also reduces the pressure on the underlying storage, thereby achieving a balance between the efficiency of uploading files to the cloud drive and the pressure on the underlying storage. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an application environment diagram of a file transfer and storage method in one embodiment;

[0025] Figure 2 This is a flowchart illustrating a file transfer and storage method in one embodiment;

[0026] Figure 3 This is an application architecture diagram of a file transfer and storage method in one embodiment;

[0027] Figure 4 This is a flowchart illustrating a small file upload method in one embodiment;

[0028] Figure 5 This is a schematic diagram of the index structure in one embodiment;

[0029] Figure 6 This is a flowchart illustrating the direct upload method in one embodiment;

[0030] Figure 7 This is a flowchart illustrating the method of transmitting data in one embodiment;

[0031] Figure 8 This is a structural block diagram of a file transfer and storage device in one embodiment;

[0032] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0034] Cloud storage is a relatively mature product, and its main function is to store local files in the cloud for purposes such as backup and sharing. Cloud storage can implement different business combinations according to different application scenarios. For example, cloud storage for enterprises can add management functions such as organizational structure and permissions; while for personal scenarios, the focus may be more on sharing control.

[0035] The primary function of cloud storage is to synchronize local files to the cloud for storage, enabling file storage, management, and sharing via remote servers. While these functions seem simple, achieving a good user experience requires attention to many details. For example, the instant upload function needs to verify the file during upload. If a matching file is found on the server, it is copied directly from the server or a file pointer is added. Therefore, cloud storage differs from local storage. To ensure data reliability and prevent data loss, distributed storage is typically used at the underlying level. Common technologies include HDFS (Hadoop Distributed File System) and Ceph. Furthermore, since cloud storage usually includes both client-side and web-based interfaces, object storage technology is also commonly used.

[0036] One challenge in cloud disk solutions is handling small files. A large number of small files can lead to the following problems: Since instant file transfers require calculating file hashes for comparison with the server, and small files are inherently small, even if the transfer succeeds, the efficiency might be lower than a direct upload. Furthermore, a large number of small files can degrade underlying write performance, as the underlying storage typically uses a dual-write strategy to ensure data reliability. Additionally, a large number of small files can lead to excessive indexes, significantly increasing the burden on metadata management, which also impacts system performance and scalability. Finally, traditional cloud disk services usually require immediate deletion of the object file in the underlying storage when deleting small files, which can lead to frequent garbage collection operations, further affecting system performance and resource utilization.

[0037] Based on this, embodiments of this application provide a file transfer and storage method to solve the problems caused by uploading small files to cloud disks and underlying storage, thereby improving the efficiency of uploading small files to cloud disks and reducing the pressure on underlying storage.

[0038] The file transfer and storage method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0039] In one exemplary embodiment, this embodiment illustrates the application of the method to a terminal. Specifically, the terminal has a cloud storage client installed, which includes, but is not limited to, mobile clients, PC (Personal Computer) clients, and Web (website service) clients. It is understood that the method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. Figure 2 As shown, in this embodiment, the method includes the following steps:

[0040] Step 202: In response to the target user's file upload instruction, determine at least one of the type and size of the file to be uploaded.

[0041] The file upload instruction can be an instruction or command to request the upload of a file to the cloud drive. The target user is the user currently requesting the file upload to the cloud drive. The file to be uploaded is the specific file to be uploaded to the cloud drive, and it can be included in the file upload instruction.

[0042] In this embodiment, when a user needs to upload a file to the cloud drive, they can do so through the cloud drive client installed on the terminal, i.e., initiate a file upload command. The client can then respond to the target user's file upload command and determine the type or size of the file to be uploaded, or determine both the type and size of the file to be uploaded simultaneously. Specifically, the file type refers to the special encoding method used by a computer to store information and identify the internally stored data. This includes, but is not limited to, text, image, and video types. For example, some file formats are used to store images, some to store programs, and some to store text information. The file type can be identified by its extension. In many operating systems, the file type is identified by its extension, which follows the file name and is separated by a period. For example, ".txt" represents a text file, ".docx" represents a Word document, and ".jpg" represents an image file, etc. The file size refers to the size of a computer file, usually expressed in bytes with a prefix.

[0043] Step 204: Determine the upload method for the file to be uploaded based on the target user's upload strategy and at least one of the type and size of the file to be uploaded.

[0044] The upload strategy can be rules or conditions pre-configured by the server for users to upload files. Upload methods include any of the following: instant upload, direct upload, and small file upload. Specifically, instant upload is a common "ignore" upload method on cloud drives, resulting in very fast file transfer speeds. Direct upload uploads the file directly to the cloud drive. Small file upload refers to uploading smaller files to the cloud drive.

[0045] In this embodiment, after the client obtains the file upload instruction from the target user, it can determine the upload method of the file to be uploaded based on the target user's upload strategy and at least one of the type and size of the file to be uploaded. That is, it determines whether the file to be uploaded to the cloud disk should be uploaded instantly, directly, or as a small file.

[0046] Step 206: Store the file to be uploaded to the server based on the upload method.

[0047] Specifically, after the client determines the specific upload method for the file to be uploaded based on the above steps, it can store the file to be uploaded on the server based on the upload method, thereby balancing the efficiency of uploading files to the cloud disk and the pressure on the underlying storage.

[0048] In the above file transfer and storage method, the client, in response to the target user's file upload command, determines at least one of the type and size of the file to be uploaded. Based on the target user's upload strategy and at least one of the type and size of the file to be uploaded, the client determines the upload method for the file to be uploaded and then stores the file to be uploaded to the server based on the upload method. This not only improves the efficiency of uploading small files to the cloud drive but also reduces the pressure on the underlying storage, thereby achieving a balance between the efficiency of uploading files to the cloud drive and the pressure on the underlying storage.

[0049] In an exemplary embodiment, in step 204, the upload method of the file to be uploaded is determined based on the target user's upload policy and at least one of the type and size of the file to be uploaded. Specifically, this may include: if the target user's upload policy specifies a first file size threshold for small file uploads, and if it is determined that the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload policy, the upload method of the file to be uploaded is determined to be small file upload.

[0050] The first file size threshold can be a condition specified in the upload strategy for determining whether to upload small files, such as 1MB (i.e., 1 megabyte) or 5MB. It is understood that the specific value of this threshold can be set based on the user's specific scenario or needs, and this embodiment does not limit this. Specifically, if the target user's upload strategy specifies a first file size threshold of 5MB for small file uploads, then when the client determines that the size of the file to be uploaded is less than or equal to 5MB, it can determine that the upload method for the file to be uploaded is small file upload.

[0051] In one scenario, if the target user's upload strategy specifies the first file type for instant upload, and the type of the file to be uploaded matches the first file type specified in the upload strategy, then the upload method for the file to be uploaded is determined to be instant upload.

[0052] The first file type can be a condition specified in the upload strategy for using instant upload. The first file type can be set based on the user's specific scenario or needs; this embodiment does not limit this. For example, if the target user's upload strategy specifies that the first file type for instant upload is a static file (such as a PDF file), then when the client determines that the file type of the file to be uploaded matches the type of a static file, if it is a PDF, then it can determine that the upload method for the file to be uploaded is instant upload.

[0053] In another scenario, if the target user's upload strategy specifies a second file type and a second file size threshold for direct upload, and the file type to be uploaded matches the second file type specified in the upload strategy, and the file size is greater than the second file size threshold, then the upload method for the file to be uploaded is determined to be direct upload. Here, the second file size threshold is greater than the first file size threshold, and the second file type can be the same as or different from the first file type.

[0054] In an exemplary embodiment, in step 202, before responding to the file upload instruction from the target user, the method may further include: in response to the login instruction from the target user, obtaining the login information of the target user, wherein the login information includes the user type and login device information of the target user; reporting the user type and login device information of the target user to the server to instruct the server to configure an upload policy for the target user based on the user type and login device information of the target user; and receiving the upload policy of the target user returned by the server.

[0055] The login instruction can be a user's request to log in to the cloud drive client. The target user's user type can be an individual user, a business user, etc. When the target user's user type is a business user, the enterprise category or nature of the target user can be further determined. The login device information can be the device type of the target user's current login device, such as a PC or mobile phone, or the operating system of the currently used device. Specifically, when a user logs in to the cloud drive client, the client can respond to the target user's login instruction, obtain the target user's login information, and then report the target user's user type and login device information to the server. This instructs the server to configure an upload policy for the target user based on the target user's user type and login device information, and then receives the target user's upload policy returned by the server. Thus, when the client receives a file upload instruction from the target user, it can determine the upload method for the file to be uploaded based on the target user's upload policy received from the server and the relevant information of the file to be uploaded, and store the file to be uploaded to the server according to the corresponding upload method.

[0056] In one embodiment, the aforementioned first file size threshold, first file type, second file type, and second file size threshold can be rules or conditions, i.e., upload policies, pre-configured by the server for the user. These upload policies can be flexibly configured based on actual needs. For example, they can be configured based on the user's type (e.g., commercial, personal use), the user's company (configurable for commercial use), and the device information of the user's device (e.g., the device's operating system). These upload policies can be configured globally, at the enterprise level, or for the individual user. When a user logs into the client, these conditions will be merged according to specified logic to form the specific user's upload policy, which will then be sent to the client.

[0057] For example, in traditional technologies, videos are typically treated as static files, and their hash values ​​(such as MD5: Message-Digest Algorithm) are calculated to determine whether instant upload is required. However, in this application, for editing equipment used by film and television production companies, video files can be defined as dynamic files, eliminating the need for instant upload checks and allowing direct upload. Similarly, videos and photos uploaded from a personal mobile phone's album can also be configured as dynamic files, eliminating the need for instant upload checks to save power and improve upload efficiency. Based on this, the above upload strategy can be configured according to specific user conditions, and the configuration process is completed on the server, without the user's awareness. By configuring the upload strategy, cloud drive uploads no longer use the default method but are instead specified by the server based on rich strategy configurations. These strategies generally save upload time and reduce the client's computational load during upload, especially on mobile terminals, reducing power consumption, saving battery life, extending terminal standby time, and thus improving user experience.

[0058] In one exemplary embodiment, the file transfer and storage method of this application can also be applied to, for example... Figure 3 The system architecture shown includes a cloud storage client and a cloud platform connected via communication. The cloud platform can further include a cloud storage management module, an object storage module, an object management module, and an indexing module. The cloud storage management module interacts with the cloud storage client to define upload policies for logged-in target users and manage file uploads. The object storage module can be cloud storage space used to store files uploaded by users. The indexing module records a business index of files stored in the cloud. The object management module allows users to manage data in the object storage and indexing modules, such as deleting and cleaning up data.

[0059] The following combination Figure 3 The system architecture shown further illustrates the file transfer and storage method of this application, such as... Figure 4As shown, taking small file upload as an example, it can specifically include:

[0060] Specifically, when a user needs to upload files, they can log in to the cloud storage client. The client then responds to the login command of the currently logged-in user (i.e., the target user), obtains the target user's login information, and reports this information to the cloud storage management module. The cloud storage management module can then configure an upload strategy for the target user based on their login information and return the upload strategy to the client.

[0061] During the file upload phase, the client can determine the appropriate upload method based on the target user's upload strategy and the file to be uploaded. Since each HTTP (Hypertext Transfer Protocol) connection requires establishing a connection and sending header information, and calculating hash values ​​like MD5 for instant file uploads also consumes computational resources, if the file size is smaller than the specified strategy threshold, the benefits of instant file uploads are considered minimal, and the instant file upload strategy can be abandoned in favor of a smaller file upload method.

[0062] In this embodiment, when the client determines that the corresponding upload method is small file upload based on the target user's upload strategy and the file to be uploaded, the specific processing logic is as follows:

[0063] The cloud storage client directly sends the file content and path of the file to be uploaded to the cloud storage management module. Upon receiving the file, the management module calculates its MD5 hash. Using the MD5 hash and file size, the management module instructs the indexing module to search for a matching file index. If the management module receives a message from the indexing module indicating a matching file index, it discards the file data to be uploaded, instructs the indexing module to create a business index for the file, and returns a successful upload result to the client, thus completing the upload and storage of the file.

[0064] If the cloud storage management module receives a message from the index module indicating that no matching file index exists, it instructs the index module to search for a large file (an aggregated object file) composed of smaller files. If the index module finds a matching aggregate object file, it returns the aggregate object to the cloud storage management module. The cloud storage management module then uses an append interface through the object storage module to instruct the object storage module to append the file to be uploaded to the specified object storage file (aggregated object file). After completing the storage, the object storage module returns a successful upload result to the cloud storage management module. The cloud storage management module inserts the business index of the file to be uploaded into the index module, i.e., inserts a file-object index. The structure of this index can be as follows: Figure 5As shown, this includes the business path of each small file, its file size, the object path of the aggregate object that each small file points to (object1 and object2 in the diagram represent different aggregate objects), and the start and end bytes of the small file within the aggregate object. The cloud storage management module can then return a successful upload result to the cloud storage client, thus completing the upload and storage of the file to be uploaded.

[0065] If the index module does not find a matching aggregated object file, it returns the search results to the cloud storage management module. The cloud storage management module then directly uploads the file to be uploaded to the object storage module. After the upload is complete, the cloud storage management module creates a business index for the file to be uploaded in the index module (the index will still record the start and end character numbers of the file to be uploaded. If other small files are uploaded later, they can be expanded based on this file until the files are merged to the specified size), and returns the upload success result to the cloud storage client, thus completing the upload and storage of the file to be uploaded.

[0066] In the above embodiments, by separating the business index from the object storage and merging small files into one object, the pressure on the object storage can be effectively reduced because the business index reads files through the object and the start and end sequence numbers of binary data.

[0067] In one exemplary embodiment, such as Figure 6 As shown, taking direct upload as an example, it can specifically include:

[0068] Specifically, when a user needs to upload files, they can log in to the cloud storage client. The client then responds to the login command of the currently logged-in user (i.e., the target user), obtains the target user's login information, and reports this information to the cloud storage management module. The cloud storage management module can then configure an upload strategy for the target user based on their login information and return the upload strategy to the client.

[0069] During the file upload phase, the client can determine the appropriate upload method based on the target user's upload strategy and the file to be uploaded. In this transmission scenario, some files, although large, are editable, so the probability of duplicates is considered low. When uploading these files, a direct upload method can be used instead of checking for instant upload, saving resources spent on MD5 calculations on the client side and improving upload efficiency.

[0070] In this embodiment, when the client determines that the corresponding upload method is direct upload based on the target user's upload policy and the file to be uploaded, the specific processing logic is as follows:

[0071] The cloud storage client sends the file size of the file to be uploaded to the cloud storage management module, requesting the necessary upload resources, such as a list of fragmented pre-signed URLs (uniform resource locator, a method used to specify the location of information on the World Wide Web service) and uploadId (upload identifier). Based on this, the cloud storage management module requests uploadIds from the object storage module. Upon receiving the uploadId from the object storage module, it calculates the fragmented upload URLs based on the uploadIds and then returns the fragmented upload URL data to the cloud storage client.

[0072] After obtaining the upload URL for a file slice, the cloud storage client slices the file to be uploaded and uploads it concurrently to the object storage module according to the configured policy. Simultaneously, it records the ETag (Entity tag, representing the entity value of the requested variable; an ETag is an identifier for a resource on the server) of each file slice. Upon receiving the sliced ​​upload data from the cloud storage client, the object storage module returns the ETag to the cloud storage client.

[0073] After the upload is complete, the cloud storage client sends the file path, ETag list, and uploadId of the file to be uploaded to the cloud storage management module, instructing the management module to pass the ETag list to the object storage module so that the object storage module can merge the fragments. Once the object storage module has merged, it sends a successful merge notification to the cloud storage management module. Upon receiving the successful merge notification, the cloud storage management module inserts the file-object index of the file to be uploaded into the index module and sends a successful upload notification to the cloud storage client, thus completing the upload and storage of the file.

[0074] In one exemplary embodiment, such as Figure 7 As shown, taking the upload method of instant upload as an example, it can specifically include:

[0075] Specifically, when a user needs to upload files, they can log in to the cloud storage client. The client then responds to the login command of the currently logged-in user (i.e., the target user), obtains the target user's login information, and reports this information to the cloud storage management module. The cloud storage management module can then configure an upload strategy for the target user based on their login information and return the upload strategy to the client.

[0076] During the file upload phase, the client can determine the appropriate upload method based on the target user's upload strategy and the file to be uploaded. For immutable files such as MP4 and PDF (the specific file type of immutable files can be determined based on user type and logged-in device information), a fast upload method can usually be used for uploading and storage to save bandwidth and transmission time.

[0077] In this embodiment, when the client determines that the corresponding upload method is instant upload based on the target user's upload policy and the file to be uploaded, the specific processing logic is as follows:

[0078] The cloud storage client calculates the MD5 value of the file to be uploaded and sends the calculated MD5 value, file size, and file path to the cloud storage management module. The cloud storage management module uses the MD5 and file size to instruct the indexing module to search for a matching file index. If the cloud storage management module receives a message from the indexing module indicating a matching file index, it instructs the indexing module to create a new index for the file to be uploaded and returns a successful upload result to the cloud storage client, thus completing the upload and storage of the file.

[0079] If the cloud storage management module receives a message from the index module indicating that no matching file index exists, it requests an uploadId from the object storage module. Upon receiving the uploadId from the object storage module, it calculates the chunked upload URL based on the uploadId and then returns the chunked upload URL data (i.e., the uploaded resource) to the cloud storage client.

[0080] This allows the cloud storage client to obtain the file slice upload URL, slice the file to be uploaded, and concurrently upload them to the object storage module according to the configured policy, while recording the ETag of each file slice. After receiving the sliced ​​upload data from the cloud storage client, the object storage module returns an ETag to the cloud storage client.

[0081] After uploading is complete, the cloud storage client sends the file path, ETag list, and uploadId of the file to be uploaded to the cloud storage management module. This instructs the cloud storage management module to pass the ETag list to the object storage module, enabling the object storage module to merge the fragments. Once the object storage module has merged, it sends a successful merge notification to the cloud storage management module. Upon receiving the successful merge notification, the cloud storage management module inserts the file-object index of the file to be uploaded into the index module and sends a successful upload notification to the cloud storage client, thus completing the upload and storage of the file.

[0082] In an exemplary embodiment, when a user needs to delete a file in the cloud, they can first access the index module through the cloud drive management module to instruct the index module to delete the index of the corresponding file, and then the object management module will clean up the specific data stored in the object storage module.

[0083] Specifically, the object management module can periodically scan (e.g., daily or weekly) the relationship between indexes in the index module and files in the object storage module to clean up files confirmed for deletion by the customer. Cleanup scenarios include: when an object is found to have no index pointing to it (i.e., it's a detached object), it can be directly deleted from the object storage module. When an object is found to be an aggregate object composed of small files, but one of the small files in the aggregate no longer has any index pointing to it, it first finds all the small file indexes pointing to that aggregate object, downloads the relevant data based on the current indexes, combines them into a new aggregate object, modifies the indexes to point to the new aggregate object, and then deletes the old aggregate object, thus completing the small file recycling and cleanup. It also scans the MD5 hash and file size of objects (these files may have been generated during direct uploads without instant upload). When multiple identical objects are found, it finds the file indexes corresponding to these objects, modifies the indexes of these files to point to one of the objects, and deletes all the remaining identical objects, thus achieving the recycling and cleanup of directly uploaded files.

[0084] Because this embodiment separates data indexing operations from object operations, the periodic data cleanup not only reduces the coupling between business logic and storage, but also reduces the pressure on object storage due to the small file upload method. Furthermore, regular data cleanup improves the utilization rate of cloud storage space. This application also refines the file upload strategy, dividing file uploads into the three different forms mentioned above, and configuring strategies according to different user scenarios. This reduces the computational load and upload time for client uploads, thereby improving the user experience.

[0085] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0086] Based on the same inventive concept, this application also provides a file transfer and storage device for implementing the file transfer and storage method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more file transfer and storage device embodiments provided below can be found in the limitations of the file transfer and storage method described above, and will not be repeated here.

[0087] In one exemplary embodiment, such as Figure 8 As shown, a file transfer and storage device is provided. The device is applied to a client and includes: a file determination module 802, an upload method determination module 804, and a storage module 806, wherein:

[0088] The file determination module 802 is used to determine at least one of the type and size of the file to be uploaded in response to the file upload instruction of the target user, wherein the file to be uploaded is carried in the file upload instruction;

[0089] The upload method determination module 804 is used to determine the upload method of the file to be uploaded based on the target user's upload strategy and at least one of the type and size of the file to be uploaded. The upload method includes any one of instant upload, direct upload, and small file upload.

[0090] Storage module 806 is used to store the file to be uploaded to the server based on the upload method.

[0091] In an exemplary embodiment, the apparatus further includes an upload policy acquisition module, configured to: in response to a login instruction from a target user, acquire login information of the target user, the login information including the target user's user type and login device information; report the target user's user type and login device information to the server to instruct the server to configure an upload policy for the target user based on the target user's user type and login device information; and receive the upload policy of the target user returned by the server.

[0092] In an exemplary embodiment, the upload method determination module is further configured to: if the target user's upload strategy specifies a first file size threshold for small file uploads, and the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload strategy, determine that the upload method of the file to be uploaded is small file upload; if the target user's upload strategy specifies a first file type for instant uploads, and the type of the file to be uploaded matches the first file type specified by the upload strategy, determine that the upload method of the file to be uploaded is instant upload; if the target user's upload strategy specifies a second file type and a second file size threshold for direct uploads, and the type of the file to be uploaded matches the second file type specified by the upload strategy, and the size of the file to be uploaded is greater than the second file size threshold, determine that the upload method of the file to be uploaded is direct upload, and the second file size threshold is greater than the first file size threshold.

[0093] In an exemplary embodiment, the storage module is further configured to: when it is determined that the upload method of the file to be uploaded is small file upload, send the file content and file path of the file to be uploaded to the server; to instruct the server to search in the object storage for a matching file based on the file content of the file to be uploaded; if a matching file is found, create a business index for the file to be uploaded and return a successful upload result; if no matching file is found, upload the file to be uploaded to the aggregate object file based on the file path, create a business index including the aggregate object file pointed to by the file to be uploaded, and return a successful upload result; and receive the successful upload result returned by the server.

[0094] In an exemplary embodiment, the storage module is further configured to: if it is determined that the upload method of the file to be uploaded is direct upload, request upload resources from the server based on the size of the file to be uploaded; and if the upload resources are received from the server, slice the file to be uploaded according to the upload resources and upload them concurrently to the server.

[0095] In an exemplary embodiment, the storage module is further configured to: determine a corresponding hash value based on the file content of the file to be uploaded when the upload method of the file to be uploaded is determined to be instant upload; send the hash value, file path, and file size of the file to be uploaded to the server; instruct the server to search for a matching file in the object storage based on the hash value and file size of the file to be uploaded; if a matching file is found, create a business index for the file to be uploaded and return a result indicating successful instant upload; if no matching file is found, return the upload resource; receive the result indicating successful instant upload returned by the server; or, upon receiving the upload resource returned by the server, slice the file to be uploaded according to the upload resource and upload it concurrently to the server.

[0096] Each module in the aforementioned file transfer and storage device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0097] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a file transfer and storage method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0098] Those skilled in the art will understand that Figure 9 The structure shown is only a block diagram of a part of the structure related to the present application and does not constitute a limitation on the computer device on which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0099] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0100] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0101] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0105] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A file transfer and storage method, characterized in that, The method is applied to a client, and the method includes: In response to a file upload instruction from a target user, at least one of the type and size of the file to be uploaded is determined, the file to be uploaded being carried in the file upload instruction; Based on the target user's upload strategy and at least one of the type and size of the file to be uploaded, the upload method of the file to be uploaded is determined, wherein the upload method includes any one of instant upload, direct upload, and small file upload; the determination of the upload method based on the target user's upload strategy and at least one of the type and size of the file to be uploaded includes: if the target user's upload strategy specifies a first file size threshold for small file upload, and the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload strategy, the upload method of the file to be uploaded is determined to be small file upload; if the target user's upload strategy specifies a first file type for instant upload, and the type of the file to be uploaded matches the first file type specified by the upload strategy, the upload method of the file to be uploaded is determined to be instant upload; if the target user's upload strategy specifies a second file type for direct upload and a second file size threshold, and the type of the file to be uploaded matches the second file type specified by the upload strategy and the size of the file to be uploaded is greater than the second file size threshold, the upload method of the file to be uploaded is determined to be direct upload, wherein the second file size threshold is greater than the first file size threshold; The file to be uploaded is stored on the server based on the upload method described above.

2. The method according to claim 1, characterized in that, Prior to responding to the file upload instruction from the target user, the method further includes: In response to a login command from a target user, the login information of the target user is obtained, including the user type and login device information of the target user; The system reports the user type and login device information of the target user to the server, instructing the server to configure an upload policy for the target user based on the user type and login device information. Receive the upload policy of the target user returned by the server.

3. The method according to claim 1, characterized in that, The step of storing the file to be uploaded to the server based on the upload method includes: If the upload method of the file to be uploaded is determined to be a small file upload, the file content and file path of the file to be uploaded are sent to the server to instruct the server to search for a matching file in the object storage based on the file content of the file to be uploaded. If a matching file is found, a business index of the file to be uploaded is created and a result of successful upload is returned. If no matching file is found, the file to be uploaded is uploaded to the aggregate object file based on the file path, and a business index including the aggregate object file pointed to by the file to be uploaded is created, and a result of successful upload is returned. Receive the upload success result returned by the server.

4. The method according to claim 1, characterized in that, The step of storing the file to be uploaded to the server based on the upload method includes: If it is determined that the upload method of the file to be uploaded is direct upload, the server is requested to upload resources based on the size of the file to be uploaded. Upon receiving the uploaded resources returned by the server, the file to be uploaded is sliced ​​according to the uploaded resources and uploaded concurrently to the server.

5. The method according to claim 1, characterized in that, The step of storing the file to be uploaded to the server based on the upload method includes: If it is determined that the upload method of the file to be uploaded is instant upload, the corresponding hash value is determined based on the file content of the file to be uploaded. Send the hash value, file path, and file size of the file to be uploaded to the server; to instruct the server to search for a matching file in the object storage based on the hash value and file size of the file to be uploaded. If a matching file is found, create a business index for the file to be uploaded and return a result indicating successful upload in seconds; if no matching file is found, return the uploaded resource. Receive the successful transmission result returned by the server; or, Upon receiving the uploaded resources returned by the server, the file to be uploaded is sliced ​​according to the uploaded resources and uploaded concurrently to the server.

6. A file transfer and storage device, characterized in that, The device is used on a client side, and the device includes: The file determination module is used to determine at least one of the type and size of the file to be uploaded in response to the file upload instruction of the target user, wherein the file to be uploaded is carried in the file upload instruction; An upload method determination module is used to determine the upload method of the file to be uploaded based on the target user's upload strategy and at least one of the type and size of the file to be uploaded. The upload method includes any one of instant upload, direct upload, and small file upload. The upload method determination module is further used to: if the target user's upload strategy specifies a first file size threshold for small file upload, determine the upload method of the file to be uploaded as small file upload if the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload strategy; if the target user's upload strategy specifies a first file type for instant upload, determine the upload method of the file to be uploaded as instant upload if the type of the file to be uploaded matches the first file type specified by the upload strategy; if the target user's upload strategy specifies a second file type for direct upload and a second file size threshold, determine the upload method of the file to be uploaded as direct upload if the type of the file to be uploaded matches the second file type specified by the upload strategy and the size of the file to be uploaded is greater than the second file size threshold, and the second file size threshold is greater than the first file size threshold. A storage module is used to store the file to be uploaded to the server based on the upload method.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images