File transmission and storage method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a file transmission and storage method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: a client determines at least one of the type and the size of a to-be-uploaded file by responding to a file uploading instruction of a target user, and determines an uploading mode of the to-be-uploaded file based on an uploading strategy of the target user and at least one of the type and the size of the to-be-uploaded file, and storing the to-be-uploaded file to the server based on the uploading mode. The efficiency of uploading the small files to the cloud disk can be improved, and the pressure of bottom storage can be reduced, so that the efficiency of uploading the files to the cloud disk and the pressure of the bottom storage are balanced.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a file transmission and storage method, device, computer equipment, readable storage medium and program product. Background Art

[0002] In modern Internet technology, cloud disk, as an important data storage service, has received extensive attention and application. Cloud disk allows users to store, manage and share files on remote servers, greatly facilitating users' digital life and work. However, with the growing demand for cloud disk storage, especially the storage of a large number of small files, cloud disk faces some technical challenges in file transmission and storage.

[0003] In traditional technology, cloud disk services usually use the default file upload and storage method. During the file upload process, if the file already exists on the server, it can be uploaded in seconds (that is, the existing file is not re-uploaded) to save bandwidth and transmission time.

[0004] However, traditional file upload and storage methods still have some problems when dealing with a large number of small files. For example, during the instant upload process, the file hash value needs to be calculated to compare data with the server, which may lead to low transmission efficiency for small files, or even worse than direct upload. Summary of the invention

[0005] Based on this, it is necessary to provide a file transmission and storage method, device, computer equipment, computer-readable storage medium and computer program product that can improve transmission efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a file transmission and storage method, which is applied to a client and includes:

[0007] In response to a file upload instruction of a target user, determining at least one of a type and a size of a file to be uploaded, wherein the file to be uploaded is carried in the file upload instruction;

[0008] Determine an upload mode for the file to be uploaded based on the upload strategy of the target user and at least one of the type and size of the file to be uploaded, wherein the upload mode includes any one of instant upload, direct upload, and small file upload;

[0009] The file to be uploaded is stored in a server based on the uploading method.

[0010] In one of the embodiments, before responding to the file upload instruction of the target user, the method also includes: responding to the login instruction of the target user, obtaining the login information of the target user, the login information including the user type and login device information of the target user; reporting the user type and login device information of the target user to the server to instruct the server to configure an upload policy for the target user according to the user type and login device information of the target user; and receiving the upload policy of the target user returned by the server.

[0011] In one of the embodiments, the uploading method of the file to be uploaded is determined based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, including: if the upload policy of the target user specifies a first file size threshold for small file uploading, and when it is determined that the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload policy, the uploading method of the file to be uploaded is determined to be small file uploading; if the upload policy of the target user specifies a first file type for instant transmission, and when it is determined that the type of the file to be uploaded matches the first file type specified by the upload policy, the uploading method of the file to be uploaded is determined to be instant transmission; if the upload policy of the target user specifies a second file type and a second file size threshold for direct upload, and when it is determined that the type of the file to be uploaded matches the second file type specified by the upload policy and the size of the file to be uploaded is greater than the second file size threshold, the uploading method of the file to be uploaded is determined to be direct upload, and the second file size threshold is greater than the first file size threshold.

[0012] In one of the embodiments, storing the file to be uploaded to the server based on the upload method includes: when it is determined that the upload method of the file to be uploaded is small file upload, sending the file content and file path of the file to be uploaded to the server; instructing the server to search whether there is a matching file in the object storage according to the file content of the file to be uploaded, if it is determined that there is a matching file, creating a business index of the file to be uploaded and returning a successful upload result; if it is determined that there is no matching file, uploading the file to be uploaded to the aggregate object file based on the file path, creating a business index including the aggregate object file pointed to by the file to be uploaded, and returning a successful upload result; receiving the successful upload result returned by the server.

[0013] In one of the embodiments, storing the file to be uploaded to the server based on the upload method includes: when it is determined that the upload method of the file to be uploaded is direct upload, requesting upload resources from the server based on the size of the file to be uploaded; and when receiving the upload resources returned by the server, slicing the file to be uploaded according to the upload resources and concurrently uploading them to the server.

[0014] In one of the embodiments, storing the file to be uploaded to the server based on the upload method includes: when it is determined that the upload method of the file to be uploaded is instant upload, determining the corresponding hash value based on the file content of the file to be uploaded; sending the hash value, file path and file size of the file to be uploaded to the server; instructing the server to search for a matching file in the object storage according to the hash value and file size of the file to be uploaded, and if it is determined that a matching file exists, creating a business index for the file to be uploaded and returning a result of successful instant upload; if it is determined that no matching file exists, returning the upload resource; receiving the result of successful instant upload returned by the server; or, when receiving the upload resource returned by the server, slicing the file to be uploaded according to the upload resource and uploading it to the server concurrently.

[0015] In a second aspect, the present application provides a file transmission and storage device, which is applied to a client and includes:

[0016] A file determination module, configured to determine at least one of a type and a size of a file to be uploaded in response to a file upload instruction of a target user, wherein the file to be uploaded is carried in the file upload instruction;

[0017] An upload mode determination module, used to determine an upload mode of the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, wherein the upload mode includes any one of instant upload, direct upload and small file upload;

[0018] A storage module is used to store the file to be uploaded to the server based on the uploading method.

[0019] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0021] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0022] In the above-mentioned file transmission and storage method, apparatus, computer equipment, computer-readable storage medium and computer program product, the client determines at least one of the type and size of the file to be uploaded by responding to the file upload instruction of the target user, and determines the upload method of the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, and then stores the file to be uploaded to the server based on the upload method. It can not only improve the efficiency of uploading small files to the cloud disk, but also reduce the pressure of the underlying storage, thereby achieving a balance between the efficiency of uploading files to the cloud disk and the pressure of the underlying storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A diagram of an application environment of a file transmission and storage method in an embodiment;

[0025] Figure 2 A schematic diagram of a process of a file transmission and storage method in one embodiment;

[0026] Figure 3 An application architecture diagram of a file transmission and storage method in an embodiment;

[0027] Figure 4 A schematic diagram of a process of uploading a small file in one embodiment;

[0028] Figure 5 A schematic diagram of an index structure in an embodiment;

[0029] Figure 6 A schematic diagram of a flow chart of a direct uploading method in an embodiment;

[0030] Figure 7 A schematic diagram of a flow chart of a wonderful transmission method in one embodiment;

[0031] Figure 8 is a structural block diagram of a file transmission and storage device in an embodiment;

[0032] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] Cloud disk is a relatively mature product, and its main business is to store local files on the cloud for backup, sharing, etc. Cloud disk can realize different business combinations according to different application scenarios. For example, cloud disk for enterprises can add management functions such as organizational structure and permissions; while cloud disk for individuals may pay more attention to sharing control, etc.

[0035] The main function of a cloud disk is to synchronize local files to the cloud for storage, so as to store, manage and share files through a remote server. These functions may seem simple, but if you want to achieve a good user experience, you need to consider many details. For example, the second-upload function requires file verification when the user uploads. If a consistent file is found on the server, the file will no longer be uploaded but will be copied directly from the server or a file pointer will be added. Therefore, the storage of a cloud disk on the cloud is also different from that on the local computer. In order to ensure data reliability and prevent data loss, the underlying storage is generally carried out using distributed storage. Common technologies include HDFS (Hadoop Distributed File System, a distributed file system) and Ceph (a distributed file system). Since cloud disks generally have a web client in addition to the client, these technologies generally also use object storage technology.

[0036] One of the difficulties in the overall solution of cloud disk is the processing of small files. Too many small files may cause the following problems: Since the file hash value needs to be calculated to compare data with the server during the second transfer, and the small files themselves are not large, even if the second transfer is successful, the performance may not be as good as direct upload. In addition, a large number of small files will reduce the underlying write performance, because the underlying storage usually uses a double write strategy to ensure data reliability. In addition, a large number of small files will lead to too many indexes, which will greatly increase the pressure on metadata management, which will also affect the performance and scalability of the system. Finally, when traditional cloud disk services handle the deletion of small files, they usually need to immediately delete the object files in the underlying storage, which may also lead to frequent execution of recycling operations, thereby affecting the performance and resource utilization of the system.

[0037] Based on this, an embodiment of the present application provides a file transfer and storage method to solve the problems caused by uploading small files to the cloud disk and the underlying storage, thereby improving the efficiency of uploading small files to the cloud disk and reducing the pressure on the underlying storage.

[0038] The file transmission and storage method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0039] In an exemplary embodiment, this embodiment uses the method applied to a terminal as an example. Specifically, a cloud disk client is installed in the terminal, and the cloud disk client includes but is not limited to a mobile client, a PC (Personal Computer) client, and a Web (website service) client. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Figure 2 As shown, in this embodiment, the method includes the following steps:

[0040] Step 202: In response to a file upload instruction from a target user, determine at least one of a type and a size of a file to be uploaded.

[0041] The file upload instruction may be an instruction or command for requesting to upload a file to the cloud disk. The target user is the user object currently requesting to upload a file to the cloud disk. The file to be uploaded is the specific file requested to be uploaded to the cloud disk, and the file to be uploaded may be carried in the file upload instruction.

[0042] In this embodiment, when a user has a need to upload a file to a cloud disk, the file can be uploaded through the cloud disk client installed in the terminal, that is, a file upload instruction is initiated, and the client can respond to the file upload instruction of the target user and determine the type or size of the file to be uploaded, or determine the type and size of the file to be uploaded. Specifically, the type of file refers to a special encoding method used by a computer to store information, which is used to identify internally stored data. Including but not limited to text type, image type, video type, etc. For example, some file formats are used to store pictures, some are used to store programs, and some are used to store text information. The type of file can be identified by the extension. In many operating systems, the file type is identified by the extension, which follows the file name and is separated by a period. For example, ".txt" represents a text file, ".docx" represents a Word document, and ".jpg" represents an image file. The size of a file refers to the size of a computer file, usually expressed in bytes with a prefix.

[0043] Step 204: Determine an upload method for the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded.

[0044] The upload policy may be a rule or condition for uploading files that the server pre-configures for the user. The upload methods include instant upload, direct upload, and small file upload. Specifically, instant upload is a common "ignore" upload method on cloud disks, which makes the file transfer speed very fast. Direct upload is a method of uploading the file to be uploaded directly to the cloud disk. Small file upload refers to the method of uploading smaller files to the cloud disk.

[0045] In this embodiment, after the client obtains the file upload instruction of the target user, it can determine the upload method of the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, that is, determine whether the file to be uploaded should be uploaded to the cloud disk by instant upload, direct upload or small file upload.

[0046] Step 206: Store the file to be uploaded in the server based on the upload method.

[0047] Specifically, after the client determines the specific uploading method of the file to be uploaded based on the above steps, the file to be uploaded can be stored on the server based on the uploading method, thereby balancing the efficiency of uploading files to the cloud disk and the pressure of the underlying storage.

[0048] In the above-mentioned file transmission and storage method, the client determines at least one of the type and size of the file to be uploaded by responding to the file upload instruction of the target user, and determines the upload method of the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, and then stores the file to be uploaded to the server based on the upload method. This can not only improve the efficiency of uploading small files to the cloud disk, but also reduce the pressure of the underlying storage, thereby achieving a balance between the efficiency of uploading files to the cloud disk and the pressure of the underlying storage.

[0049] In an exemplary embodiment, in step 204, based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, the upload method of the file to be uploaded is determined, which may specifically include: if the upload policy of the target user specifies a first file size threshold for small file upload, when it is determined that the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload policy, determining that the upload method of the file to be uploaded is small file upload.

[0050] Among them, the first file size threshold can be a judgment condition for using small file upload specified in the upload policy, such as being set to 1MB (i.e., 1 megabyte), 5MB, etc. It is understandable that the specific size of the threshold can be set based on the specific scenario or needs of the user, and this embodiment does not limit this. Specifically, if the first file size threshold for small file upload is specified as 5MB in the upload policy of the target user, then when the client determines that the size of the file to be uploaded is less than 5MB or equal to 5MB, it can determine that the upload method of the file to be uploaded is small file upload.

[0051] In one scenario, if the target user's upload policy specifies a first file type that is transferred in seconds, when it is determined that the type of the file to be uploaded matches the first file type specified in the upload policy, the upload mode of the file to be uploaded is determined to be transferred in seconds.

[0052] Among them, the first file type can be a judgment condition specified in the upload policy when uploading in seconds, and the first file type can be set based on the specific scenario or needs of the user, and this embodiment does not limit this. For example, if the target user's upload policy specifies that the first file type when uploading in seconds is a static file (such as a PDF type file), when the client determines that the file type of the file to be uploaded matches the type of the static file, if it is a PDF type, it can determine that the upload method of the file to be uploaded is seconds.

[0053] In another scenario, if the target user's upload policy specifies a second file type and a second file size threshold for direct upload, when it is determined that the type of the file to be uploaded matches the second file type specified by the upload policy, and the size of the file to be uploaded is greater than the second file size threshold, it is determined that the upload method of the file to be uploaded is direct upload. The second file size threshold is greater than the first file size threshold, and the second file type may be the same as the first file type or different from the first file type.

[0054] In an exemplary embodiment, in step 202, before responding to the file upload instruction of the target user, the above method may also include: responding to the login instruction of the target user, obtaining the login information of the target user, wherein the login information includes the user type and login device information of the target user; reporting the user type and login device information of the target user to the server to instruct the server to configure an upload policy for the target user according to the user type and login device information of the target user; and receiving the upload policy of the target user returned by the server.

[0055] Among them, the login instruction can be an instruction or command for the user to request to log in to the cloud disk client. The user type of the target user can be a personal user, a commercial user, etc. When the user type of the target user is a commercial user, the enterprise category or enterprise nature to which the target user belongs can be further determined. The login device information can be the device type of the device currently used by the target user to log in, such as PC, mobile phone, etc., or the operating system of the currently used device, etc. Specifically, when the user logs in to the cloud disk client, the client can respond to the login instruction of the target user, obtain the login information of the target user, and then report the user type and login device information of the target user to the server. In order to instruct the server to configure an upload policy for the target user according to the user type and login device information of the target user, and then receive the upload policy of the target user returned by the server. Therefore, when the client receives the file upload instruction of the target user, it can determine the upload method of the file to be uploaded based on the upload policy of the target user received from the server and the relevant information of the file to be uploaded, and store the file to be uploaded to the server according to the corresponding upload method.

[0056] In one embodiment, the first file size threshold, the first file type, the second file type, and the second file size threshold may be rules or conditions for uploading files that are pre-configured by the server for the user, i.e., upload policies. These upload policies can be flexibly configured based on actual needs. For example, they can be configured based on the type of user (such as commercial use, personal use, etc.), the company to which the user belongs (configurable for commercial use), and the device information of the device used by the user (such as the operating system of the device, etc.) as reference conditions. These upload policies can be configured globally, for the enterprise, or for the user himself. When the user logs in to the client, these conditions will be merged according to the specified logic to form an upload policy for the specific user and sent to the client.

[0057] For example, in traditional technology, videos are usually treated as static files, and the hash value (such as MD5: Message-Digest Algorithm) is calculated to determine whether it needs to be transmitted in seconds. In this application, for the editing equipment of film and television production companies, video files can be defined as dynamic files, and there is no need to perform a second transmission judgment when uploading, but directly upload. Similarly, in personal mobile phones, videos and photos uploaded from the album can also be configured as dynamic files, without the need for a second transmission judgment to save computing power and improve upload efficiency. Based on this, the above upload strategy can be configured based on the user's specific reference conditions, and the configuration process is completed on the server without the user's perception. Through the configuration of the upload strategy, the upload of the cloud disk is no longer uploaded in the default way, but is specified by the server based on a rich policy configuration. According to these policies, the upload time is saved overall, and the client's calculation amount during upload is reduced, especially in mobile terminals, which reduces computing energy consumption, is conducive to saving power, and extends the standby time of the terminal, thereby improving the user experience.

[0058] In an exemplary embodiment, the file transmission and storage method of the present application can also be applied to Figure 3 In the system architecture shown. The architecture includes a network disk client and a cloud that are communicatively connected, wherein the cloud may also include a network disk management module, an object storage module, an object management module, and an index module. The network disk management module can interact with the network disk client to formulate an upload strategy for the logged-in target user and manage file uploads. The object storage module can be a storage space in the cloud for storing files uploaded by users to the cloud. The index module is used to record the business index of files stored in the cloud. The object management module user manages the data in the object storage module and the index module, such as deletion and cleaning.

[0059] The following combination Figure 3 The system architecture shown further illustrates the file transmission and storage method of the present application, such as Figure 4As shown, taking the upload method of small file upload as an example, it may specifically include:

[0060] Specifically, when a user has a need to upload a file, he can log in to the cloud disk client, and the client responds to the login instruction of the current logged-in user, that is, the target user, obtains the login information of the target user, and reports the login information of the target user to the cloud network disk management module. The network disk management module can configure an upload policy for the target user according to the login information of the target user, and return the upload policy of the target user to the client.

[0061] Later, during the file upload phase, the client can determine the corresponding upload method based on the target user's upload strategy and the file to be uploaded. Since HTTP (Hypertext Transfer Protocol) requires the creation of a connection and the sending of header information for each connection, and the calculation of hash values ​​such as MD5 by second-upload also consumes computing resources. Therefore, when the file is smaller than the specified policy setting value, it is considered that the benefit of second-upload is not high, so the second-upload strategy can be abandoned and the small file upload method can be adopted.

[0062] In this embodiment, when the client determines that the corresponding upload mode is small file upload based on the upload policy of the target user and the file to be uploaded, the specific processing logic is as follows:

[0063] The network disk client directly sends the file content and file path of the file to be uploaded to the network disk management module. After receiving the file, the network disk management module calculates the MD5 value of the file. The network disk management module uses MD5 and the file size to instruct the index module to find out whether there is a matching file index. If the network disk management module receives a message from the index module that there is a matching file index, the file data of the file to be uploaded is discarded, and the index module is instructed to create a business index for the file to be uploaded, and the result of successful upload is returned to the network disk client, thereby completing the upload and storage of the file to be uploaded.

[0064] If the network disk management module receives a message from the index module indicating that there is no matching file index, it instructs the index module to look for a large file that is an integration of small files, i.e., an aggregate object file. If the index module finds a matching aggregate object file, it returns the aggregate object to the network disk management module. The network disk management module then instructs the object storage module to append the file to be uploaded to the object storage file specified by the object storage module, i.e., the aggregate object file, through the object storage module append interface. After the object storage module completes the storage, it returns the result of successful upload to the network disk management module. The network disk management module inserts the business index of the file to be uploaded into the index module, i.e., inserts the file-object index. The structure of the index can be as follows: Figure 5As shown, it includes the business path of each small file, the file size, the object path of the aggregate object pointed to by each small file (object1 and object2 in the figure represent different aggregate objects), and the start and end bytes of the small file in the aggregate object. Then, the network disk management module can return the result of successful upload to the network disk client, thereby completing the upload and storage of the files to be uploaded.

[0065] If the index module does not find a matching aggregate object file, it returns the search result to the network disk management module, and the network disk management module directly uploads the file to be uploaded to the object storage module. After the upload is completed, the network disk management module creates a business index for the file to be uploaded in the index module (the index will still record the start and end character numbers of the file to be uploaded. If other small files are uploaded later, they can also be expanded based on this file until the files are merged to the specified size), and returns the result of successful upload to the network disk client, thereby completing the upload and storage of the file to be uploaded.

[0066] In the above embodiment, by separating the business index from the object storage, small files are merged into one object. Since the business index reads the file through the object and the binary data start and end numbers, the pressure on the object storage can be effectively reduced.

[0067] In an exemplary embodiment, Figure 6 As shown, taking direct upload as an example, the upload method may specifically include:

[0068] Specifically, when a user has a need to upload a file, he can log in to the cloud disk client, and the client responds to the login instruction of the current logged-in user, that is, the target user, obtains the login information of the target user, and reports the login information of the target user to the cloud network disk management module. The network disk management module can configure an upload policy for the target user according to the login information of the target user, and return the upload policy of the target user to the client.

[0069] Later, during the file upload phase, the client can determine the corresponding upload method based on the target user's upload policy and the file to be uploaded. In the transmission scenario, although some files are large, they are editable files, so it can be considered that the probability of these files being duplicated is not high. When uploading these files, there is no need to judge the second transmission, but to use the direct upload method to save the resources of MD5 calculation on the client and improve the upload efficiency.

[0070] In this embodiment, when the client determines that the corresponding upload method is direct upload based on the upload policy of the target user and the file to be uploaded, the specific processing logic is as follows:

[0071] The network disk client transmits the file size to be uploaded to the network disk management module, and applies to it for the upload resources required for uploading, such as the list of segment pre-signed URLs (uniform resource locator, a method used to specify information locations on the World Wide Web service program of the Internet) and uploadId (upload identifier). Based on this, the network disk management module applies for uploadId from the object storage module, and after receiving the uploadId returned by the object storage module, it calculates the segment upload URL based on the uploadId, and then returns the segment upload URL data to the network disk client.

[0072] After obtaining the slice upload URL, the network disk client slices the file to be uploaded and uploads it to the object storage module concurrently according to the configuration policy, while recording the Etag (Entity tag abbreviation, indicating the entity value of the requested variable, Etag is the identifier of a resource on the server side) of the file slice. After receiving the sliced ​​data uploaded by the network disk client, the object storage module returns the Etag to the network disk client.

[0073] After the upload is completed, the network disk client passes the file path, Etag list and uploadId of the file to be uploaded to the network disk management module, and instructs the network disk management module to pass the Etag list to the object storage module so that the object storage module can merge the shards. After the object storage module completes the merger, it returns a successful merger notification to the network disk management module. After receiving the successful merger notification, the network disk management module inserts the file-object index of the file to be uploaded into the index module, and returns a successful upload notification to the network disk client, thereby completing the upload and storage of the file to be uploaded.

[0074] In an exemplary embodiment, Figure 7 As shown, taking the upload mode of instant upload as an example, it may specifically include:

[0075] Specifically, when a user has a need to upload a file, he can log in to the cloud disk client, and the client responds to the login instruction of the current logged-in user, that is, the target user, obtains the login information of the target user, and reports the login information of the target user to the cloud network disk management module. The network disk management module can configure an upload policy for the target user according to the login information of the target user, and return the upload policy of the target user to the client.

[0076] Later, during the file upload phase, the client can determine the corresponding upload method based on the target user's upload policy and the file to be uploaded. For some immutable files such as MP4, PDF, etc. (the specific file type of immutable files can be determined based on user type and login device information, etc.), in order to save bandwidth and transmission time, they can usually be uploaded and stored in seconds.

[0077] In this embodiment, when the client determines that the corresponding upload mode is instant upload based on the upload policy of the target user and the file to be uploaded, the specific processing logic is as follows:

[0078] The network disk client calculates the MD5 value of the file to be uploaded, and passes the calculated MD5 value, file size, file path, etc. to the network disk management module. The network disk management module instructs the index module to find out whether there is a matching file index based on the MD5 and file size. If the network disk management module receives a message from the index module that there is a matching file index, it instructs the index module to create a new index for the file to be uploaded and returns the result of successful upload to the network disk client, thereby completing the upload and storage of the file to be uploaded.

[0079] If the network disk management module receives a message from the index module that there is no matching file index, it applies for uploadId from the object storage module, and after receiving the uploadId returned by the object storage module, it calculates the multi-segment upload URL based on the uploadId, and then returns the multi-segment upload URL data (i.e., uploaded resources) to the network disk client.

[0080] After obtaining the slice upload URL, the network disk client will slice the file to be uploaded, and upload it to the object storage module concurrently according to the configuration strategy, while recording the Etag of the file slice. After receiving the sliced ​​data uploaded by the network disk client, the object storage module returns the Etag to the network disk client.

[0081] After the upload is completed, the network disk client passes the file path, Etag list and uploadId of the file to be uploaded to the network disk management module to instruct the network disk management module to pass the Etag list to the object storage module so that the object storage module can merge the shards. After the object storage module completes the merger, it returns a successful merger notification to the network disk management module. After receiving the successful merger notification, the network disk management module inserts the file-object index of the file to be uploaded into the index module, and returns a successful upload notification to the network disk client, thereby completing the upload and storage of the file to be uploaded.

[0082] In an exemplary embodiment, when a user needs to delete a file in the cloud, he can first access the index module through the network disk management module to instruct the index module to delete the index of the corresponding file, and then the object management module cleans up the specific data stored in the object storage module.

[0083] Specifically, the object management module can periodically scan (such as daily or weekly scans) the relationship between the index in the index module and the file in the object storage module to clean up the files that the customer confirms to delete. The cleaning scenarios include: when it is found that an object has no index pointing to it, that is, the object is a free object, so the object can be directly deleted from the object storage module. When it is found that an object is an aggregate object composed of small files, but a small file part in the combination has no index pointing to it, then all small file indexes pointing to the aggregate object can be found first, and the relevant data can be downloaded according to the current index to form a new aggregate object, and the index can be modified to point to the new aggregate object, and then the old aggregate object can be deleted, so as to complete the recovery and cleaning of small files. Scan the MD5 and file size of the object (these files may be generated by direct upload without instant upload). When it is found that there are multiple consistent objects, find the file indexes corresponding to these objects, modify the indexes of these files to point to one of the objects, and delete all the remaining identical objects to achieve the recovery and cleaning of directly uploaded files.

[0084] Since the data index operation is separated from the object operation in this embodiment, the timed recovery operation not only reduces the coupling between business and storage, but also reduces the pressure of object storage based on the small file upload method. In addition, by regularly cleaning up data, the utilization rate of cloud storage space can be improved. This application also refines the file upload strategy, divides file upload into the above three different forms, and configures the strategy according to different user scenarios, which can also reduce the computational workload and upload time of client uploads, and improve user experience.

[0085] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0086] Based on the same inventive concept, the embodiment of the present application also provides a file transmission and storage device for implementing the file transmission and storage method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file transmission and storage device embodiments provided below can refer to the limitations on the file transmission and storage method above, and will not be repeated here.

[0087] In an exemplary embodiment, Figure 8 As shown, a file transmission and storage device is provided, which is applied to a client and includes: a file determination module 802, an upload mode determination module 804 and a storage module 806, wherein:

[0088] A file determination module 802, configured to determine at least one of a type and a size of a file to be uploaded in response to a file upload instruction of a target user, wherein the file to be uploaded is carried in the file upload instruction;

[0089] An upload mode determination module 804, configured to determine an upload mode of the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, wherein the upload mode includes any one of instant upload, direct upload, and small file upload;

[0090] The storage module 806 is used to store the file to be uploaded to the server based on the uploading method.

[0091] In an exemplary embodiment, the device also includes an upload policy acquisition module, which is used to: obtain the login information of the target user in response to the login instruction of the target user, the login information including the user type and login device information of the target user; report the user type and login device information of the target user to the server to instruct the server to configure an upload policy for the target user based on the user type and login device information of the target user; and receive the upload policy of the target user returned by the server.

[0092] In an exemplary embodiment, the upload mode determination module is also used for: if the upload policy of the target user specifies a first file size threshold for small file upload, when it is determined that the size of the file to be uploaded is less than or equal to the first file size threshold specified by the upload policy, determining that the upload mode of the file to be uploaded is small file upload; if the upload policy of the target user specifies a first file type that can be uploaded in seconds, when it is determined that the type of the file to be uploaded matches the first file type specified by the upload policy, determining that the upload mode of the file to be uploaded is uploaded in seconds; if the upload policy of the target user specifies a second file type and a second file size threshold for direct upload, when it is determined that the type of the file to be uploaded matches the second file type specified by the upload policy and the size of the file to be uploaded is greater than the second file size threshold, determining that the upload mode of the file to be uploaded is direct upload, and the second file size threshold is greater than the first file size threshold.

[0093] In an exemplary embodiment, the storage module is also used to: when it is determined that the upload method of the file to be uploaded is small file upload, send the file content and file path of the file to be uploaded to the server; to instruct the server to search whether there is a matching file in the object storage according to the file content of the file to be uploaded, if it is determined that there is a matching file, create a business index of the file to be uploaded, and return a successful upload result; if it is determined that there is no matching file, upload the file to be uploaded to the aggregate object file based on the file path, create a business index including the aggregate object file pointed to by the file to be uploaded, and return a successful upload result; receive the successful upload result returned by the server.

[0094] In an exemplary embodiment, the storage module is also used to: when it is determined that the upload method of the file to be uploaded is direct upload, request upload resources from the server based on the size of the file to be uploaded; when receiving the upload resources returned by the server, slice the file to be uploaded according to the upload resources and upload them to the server concurrently.

[0095] In an exemplary embodiment, the storage module is also used to: when it is determined that the upload mode of the file to be uploaded is instant upload, determine the corresponding hash value based on the file content of the file to be uploaded; send the hash value, file path and file size of the file to be uploaded to the server; to instruct the server to search whether there is a matching file in the object storage according to the hash value and file size of the file to be uploaded. If it is determined that there is a matching file, create a business index for the file to be uploaded and return a result of successful instant upload; if it is determined that there is no matching file, return the upload resource; receive the result of successful instant upload returned by the server; or, when receiving the upload resource returned by the server, slice the file to be uploaded according to the upload resource and upload it to the server concurrently.

[0096] Each module in the above-mentioned file transmission and storage device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0097] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a file transfer and storage method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0098] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0099] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0100] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0101] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0103] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0105] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file transmission and storage method, characterized in that: The method is applied to a client, and the method comprises: In response to a file upload instruction of a target user, determining at least one of a type and a size of a file to be uploaded, wherein the file to be uploaded is carried in the file upload instruction; Determine an upload mode for the file to be uploaded based on the upload strategy of the target user and at least one of the type and size of the file to be uploaded, wherein the upload mode includes any one of instant upload, direct upload and small file upload; The file to be uploaded is stored in a server based on the uploading method.

2. The method according to claim 1, characterized in that Before responding to the target user's file upload instruction, the method further includes: In response to a login instruction of a target user, acquiring login information of the target user, the login information including a user type and login device information of the target user; Reporting the user type and login device information of the target user to the server to instruct the server to configure an upload policy for the target user according to the user type and login device information of the target user; Receive the upload policy of the target user returned by the server.

3. The method according to claim 1, characterized in that: The determining the uploading method of the file to be uploaded based on the upload strategy of the target user and at least one of the type and size of the file to be uploaded includes: If a first file size threshold for small file upload is specified in the upload policy of the target user, when it is determined that the size of the file to be uploaded is less than or equal to the first file size threshold specified in the upload policy, determining that the upload mode of the file to be uploaded is small file upload; If the target user's upload policy specifies a first file type that can be uploaded in seconds, when it is determined that the type of the file to be uploaded matches the first file type specified by the upload policy, determining that the upload mode of the file to be uploaded is to be uploaded in seconds; If a second file type and a second file size threshold for direct upload are specified in the upload policy of the target user, when it is determined that the type of the file to be uploaded matches the second file type specified by the upload policy and the size of the file to be uploaded is greater than the second file size threshold, it is determined that the upload method of the file to be uploaded is direct upload, and the second file size threshold is greater than the first file size threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The storing the to-be-uploaded file to the server based on the uploading method includes: In the case where it is determined that the upload mode of the file to be uploaded is small file upload, the file content and file path of the file to be uploaded are sent to the server; to instruct the server to search whether there is a matching file in the object storage according to the file content of the file to be uploaded, and if it is determined that there is a matching file, create a business index of the file to be uploaded and return a result of successful upload; if it is determined that there is no matching file, upload the file to be uploaded to the aggregate object file based on the file path, create a business index including the aggregate object file pointed to by the file to be uploaded, and return a result of successful upload; Receive a successful upload result returned by the server.

5. The method according to any one of claims 1 to 3, characterized in that: The storing the to-be-uploaded file to the server based on the uploading method includes: When it is determined that the uploading method of the file to be uploaded is direct uploading, requesting the server to upload resources based on the size of the file to be uploaded; When receiving the upload resource returned by the server, the file to be uploaded is sliced ​​according to the upload resource and uploaded to the server concurrently.

6. The method according to any one of claims 1 to 3, characterized in that: The storing the to-be-uploaded file to the server based on the uploading method includes: When it is determined that the upload mode of the file to be uploaded is instant upload, determining a corresponding hash value based on the file content of the file to be uploaded; Send the hash value, file path and file size of the file to be uploaded to the server; instruct the server to search for a matching file in the object storage according to the hash value and file size of the file to be uploaded, and if it is determined that there is a matching file, create a business index for the file to be uploaded and return a result of successful upload; if it is determined that there is no matching file, return the upload resource; Receive a result indicating successful instant transmission returned by the server; or, When receiving the upload resource returned by the server, the file to be uploaded is sliced ​​according to the upload resource and uploaded to the server concurrently.

7. A file transmission and storage device, characterized in that: The device is applied to a client, and comprises: A file determination module, configured to determine at least one of a type and a size of a file to be uploaded in response to a file upload instruction of a target user, wherein the file to be uploaded is carried in the file upload instruction; An upload mode determination module, used to determine an upload mode of the file to be uploaded based on the upload policy of the target user and at least one of the type and size of the file to be uploaded, wherein the upload mode includes any one of instant upload, direct upload and small file upload; A storage module is used to store the file to be uploaded to the server based on the uploading method.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images