File uploading method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a file uploading method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: pre-fragmenting a local cache space based on a file fragmentation rule of object storage; receiving file data sent by the client, and caching the file data according to a fragment cache region obtained by pre-fragmentation; uploading the file data cached in the cache space to an object storage server, retaining the file data cached in the fragmented hot area, and performing overlay storage under the condition that the subsequently received file data needs to be overlay-stored in the non-fragmented hot area; and when the file data cached in the fragmented hot area needs to be modified, modifying the file data cached in the fragmented hot area, and uploading the file data cached in the fragmented hot area to an object storage server so as to indicate the object storage server to perform merging based on the received file data. By adopting the method, the file uploading efficiency can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of file storage technology, and in particular to a file uploading method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] With the continuous development of modern computer network technology, object storage file gateway is an important data storage method. It allows users to interact with object storage conveniently by providing a standard file system interface. In the application scenario of object storage file gateway, the upload process of large files itself faces a huge challenge of time cost. What is more complicated is that when large files need to be read-modify-write (RMW) operations at the same time during the upload process, the double burden significantly increases processing time and resource consumption.

[0003] In the related art, when an RMW request occurs during the multi-segment upload process, if the part hit by the request does not exist in the cache, it is necessary to first upload all the continuous segments in the cache and then perform a completion operation, then read the part that needs to be modified into the cache for modification, and then create a new segment to upload, and then complete the related RMW request. In the process of constantly creating multi-segment upload operations and performing a large number of uploads, a large number of repeated object data operations will be generated, resulting in a slow file upload rate. Summary of the invention

[0004] Based on this, it is necessary to provide a file uploading method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the file uploading rate in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for uploading a file, comprising:

[0006] Pre-shard the local cache space based on the file sharding rules of object storage;

[0007] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0008] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0009] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0010] In one embodiment, the method further comprises:

[0011] During the process of file data transmission, if there is file data cached in the target non-shard hot zone that needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, the corresponding upload bits of the other file data are filled based on preset characters, and the filled results are uploaded to the object storage server together with the file data cached in the target non-shard hot zone.

[0012] In one embodiment, the method further comprises:

[0013] In the case where the file data cached in the target non-shard hot zone needs to be modified, the file data cached in the target non-shard hot zone is downloaded from the object storage server, modified, and the modified file data is uploaded to the object storage server.

[0014] In one embodiment, the division method of the shard hot zone includes:

[0015] According to the historical modification frequency of the file data in the file to be stored in the object, determine the size of the target file data to be stored in the shard hot zone and the offset in the file;

[0016] According to the size of the target file data and the offset of the target file data in the file, the cache space is divided into shard hot zones.

[0017] In one embodiment, uploading the file data cached in the cache space to the object storage server includes:

[0018] When the cache space is full, the file data cached in the cache space is uploaded to the object storage server.

[0019] In one embodiment, determining the size of target file data to be stored in the shard hot zone and the offset in the file includes:

[0020] Determine the size and position number of the target file data in the shard cache area according to the shard cache area obtained by pre-sharding;

[0021] The position number is subtracted by one, and then multiplied by the size of the target file data to obtain the offset of the target file data.

[0022] In a second aspect, the present application also provides a file uploading device, comprising:

[0023] The sharding module is used to pre-shard the local cache space based on the file sharding rules of object storage;

[0024] A receiving module, used for receiving file data sent by a client, and caching the file data according to a shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0025] A storage module is used to upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone;

[0026] The upload module is used to modify the file data cached in the shard hot zone when it is necessary to modify the file data cached in the shard hot zone during the file data transmission process, and upload the file data cached in the shard hot zone to the object storage server to instruct the object storage server to merge based on the received file data.

[0027] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0028] Pre-shard the local cache space based on the file sharding rules of object storage;

[0029] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0030] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0031] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Pre-shard the local cache space based on the file sharding rules of object storage;

[0034] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0035] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0036] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0037] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0038] Pre-shard the local cache space based on the file sharding rules of object storage;

[0039] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0040] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0041] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0042] The above-mentioned file uploading method, device, computer equipment, computer-readable storage medium and computer program product first pre-slice the local cache space based on the file slicing rules of the object storage; receive the file data sent by the client, and cache the file data according to the slicing cache area obtained by pre-slicing; upload the file data cached in the cache space to the object storage server, retain the file data cached in the slicing hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-slicing hot zone; during the file data transmission process, when it is necessary to modify the file data cached in the slicing hot zone, modify the file data cached in the slicing hot zone until it is no longer necessary to modify the file data cached in the slicing hot zone, and upload the file data cached in the slicing hot zone to the object storage server to instruct the object storage server to merge based on the received file data. In this way, by pre-slicing and setting the slicing hot zone, the large amount of upload overhead caused by the need to modify the file is reduced, and the file upload efficiency is accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 An application environment diagram of a file uploading method in an embodiment;

[0045] Figure 2 A schematic diagram of a process of uploading a file in one embodiment;

[0046] Figure 3 A schematic diagram of an object storage server for uploading file data in one embodiment;

[0047] Figure 4 A schematic diagram of modifying file data and uploading it to an object storage server in one embodiment;

[0048] Figure 5 It is a structural block diagram of a file uploading device in one embodiment;

[0049] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The file upload method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the file gateway 102 communicates with the object storage server 104 through the network. The data storage system can store data that the object storage server 104 needs to process. The data storage system can be integrated on the object storage server 104, or it can be placed on the cloud or other network servers.

[0052] The client determines the file data that needs to be uploaded and writes the file data into the cache space in the file gateway 102. After receiving the file data uploaded by the client, the file gateway 102 uploads the file data to the object storage server. After receiving all the file data, the object storage server merges the received file data to obtain the file data that needs to be uploaded.

[0053] The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The object storage server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a file upload method is provided, which is applied to Figure 1 The file gateway 102 in the example is used as an example to illustrate, including the following steps 202 to 208. Among them:

[0055] Step 202: pre-slice the local cache space based on the file slicing rule of the object storage.

[0056] Among them, object storage is a data storage architecture that stores data in the form of objects in a storage system.

[0057] Exemplarily, the file gateway pre-slices the local cache space according to the file slicing rules of the object storage. The size and position of the pre-slice can be set according to the type of specific file data, and the embodiment of the present application does not limit this.

[0058] Step 204: Receive the file data sent by the client, and cache the file data according to the slice cache area obtained by pre-slicing.

[0059] The shard cache area includes a shard hot zone and a non-shard hot zone. The file data may be a video package file, a picture file or a word document, or may be file data in other formats, which is not limited in the present embodiment.

[0060] Optionally, the file gateway gradually receives file data sent by the client, and stores the files in the shard cache area obtained according to pre-sharding. The shard cache area includes a shard hot zone and a non-shard hot zone. The file data cached in the shard hot zone will be stored in the cache space before the file transfer is completed; the file data in the non-shard hot zone will be overwritten by subsequent file data after being uploaded to the object storage server.

[0061] Step 206, upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone.

[0062] Exemplarily, the file data cached in the cache space is uploaded to the object storage server. When the file data is subsequently received, the subsequently received file data overwrites the file data in the previous non-sharded hot zone, and the overwritten file data is uploaded to the object storage server. For the file data in the sharded hot zone, no overwriting operation is performed.

[0063] Step 208, during the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified, and the file data cached in the shard hot zone is uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0064] Optionally, before the object storage server has received all the file data, when it is necessary to modify the file data cached in the shard hot zone, the file data cached in the shard hot zone is directly modified accordingly, and the modified file data cached in the shard hot zone is uploaded to the object storage server. After all modifications to the file data are completed and the object storage server has received all the file data, the object storage server is instructed to merge all the file data to complete the file upload.

[0065] In some embodiments, during the file transfer process, the file data cached in the shard hot zone may be modified multiple times. After the multiple modifications are completed and the object storage server receives all the file data, the object storage server is instructed to merge all the file data to complete the file upload.

[0066] In the above-mentioned file upload method, the local cache space is pre-sliced ​​based on the file slicing rules of the object storage; the file data sent by the client is received, and the file data is cached according to the slicing cache area obtained by pre-slicing; the file data cached in the cache space is uploaded to the object storage server, and the file data cached in the slicing hot zone is retained. When the file data received subsequently needs to be overwritten and stored in the non-slicing hot zone, the overwritten storage is performed; during the file data transmission process, when the file data cached in the slicing hot zone needs to be modified, the file data cached in the slicing hot zone is modified until the file data cached in the slicing hot zone no longer needs to be modified, and the file data cached in the slicing hot zone is uploaded to the object storage server to instruct the object storage server to merge based on the received file data. In this way, by pre-slicing and setting the slicing hot zone, a large amount of upload overhead caused by the need to modify files is reduced, and the file upload efficiency is accelerated.

[0067] In an exemplary embodiment, the file upload method also includes: during the process of file data transmission, if there is file data cached in the target non-sharded hot zone that needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, the corresponding upload bits of the other file data are filled based on preset characters, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

[0068] In actual implementation, during the process of file data transmission, when the file data cached in the target non-sharded hot zone needs to be uploaded to the object server together, but other file data is not cached in the cache space, the preset characters are used to fill in the corresponding upload bits of other file data, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

[0069] For example, during the file transfer process, ABCDE file data needs to be uploaded to the object storage server, but only AD data is cached in the cache space, with a total of 5 upload bits. At this time, the remaining upload bits can be filled with 0 and uploaded together, that is, A00D0 file data is uploaded to the object storage server.

[0070] In the above embodiment, when uploading is required, preset characters are used to fill the upload position that is not cached in the cache space and then the upload is performed, thereby avoiding the situation where file data is overwritten due to untimely uploading, which ultimately leads to the loss of file data.

[0071] In an exemplary embodiment, the file upload method also includes: when it is necessary to modify the file data cached in the target non-sharded hot zone, downloading the file data cached in the target non-sharded hot zone from the object storage server, modifying it, and uploading the modified file data to the object storage server.

[0072] In actual implementation, when the file data in the cache space needs to be modified and the modified file data is located in the target non-sharded hot zone, the file data cached in the target non-sharded hot zone is downloaded from the object storage server and modified, and the modified file data is uploaded to the object storage server.

[0073] Exemplarily, when A in the target non-shard hot zone needs to be modified, the A00D0 file data is downloaded from the object storage server, A is modified to obtain A', and then the A'00D0 file data is uploaded to the object storage server.

[0074] In the above embodiment, by downloading the corresponding file data and modifying it, and then uploading it again after the modification, a faster file data modification operation is achieved, thereby improving the file upload rate.

[0075] In an exemplary embodiment, the method of dividing the shard hot zone includes: determining the size of the target file data to be stored in the shard hot zone and the offset in the file according to the historical modification frequency of the file data in the file to be object stored; and dividing the shard hot zone in the cache space according to the size of the target file data and the offset of the target file data in the file.

[0076] In actual implementation, the format and usage scenario of the file data in the file to be stored in the object are obtained, and the size, offset and modification frequency of each file data in the file data are determined according to the format and usage scenario of the file data. When the modification frequency of a certain file data exceeds the preset number of times, the cache space is divided into shard hot zones according to the size and offset corresponding to the corresponding file data.

[0077] In the above embodiment, by presetting the shard hot zone, when the file data needs to be modified, the corresponding modification can be directly performed and then uploaded, which reduces the overhead required for the modification process and improves the performance of file upload.

[0078] In an exemplary embodiment, uploading the file data cached in the cache space to the object storage server includes: when the cache space is full, uploading the file data cached in the cache space to the object storage server.

[0079] In actual implementation, when the cache space is full, the file data cached in the cache space is uploaded to the object storage server.

[0080] In another embodiment, when the file data in the cache space meets the preset condition, the file data cached in the cache space is uploaded to the object storage server. The preset condition may be uploading the file data cached in the cache space to the object storage server once every 30 seconds, or other conditions, which may be set by the user according to actual conditions, and this application does not limit this.

[0081] In the above embodiment, when the cache space is full, the file data in the cache space is uploaded to the object storage server, thereby preventing the possibility that the subsequently uploaded file data will overwrite the existing file data.

[0082] In an exemplary embodiment, the size of target file data to be stored in a shard hot zone and its offset in the file are determined, including: determining the size and position number of the target file data in the shard cache area based on the shard cache area obtained by pre-sharding; subtracting one from the position number and multiplying it by the size of the target file data to obtain the offset of the target file data.

[0083] In actual implementation, the size and position number of the target file data in the shard cache area are determined based on the shard cache area obtained by pre-sharding and the target file data; the position number is subtracted by one and multiplied by the size of the target file data to obtain the offset of the target file data.

[0084] In the above embodiment, by determining the offset of the file data, concurrent uploading according to the offset can be achieved, thereby accelerating the upload rate of the file data.

[0085] To explain the file upload method in the present application in detail, an embodiment is used for explanation below. By way of example, the present application explains the file upload method in a specific file upload scenario.

[0086] First, the file gateway pre-shards the local cache space according to the file sharding rules of the object storage.

[0087] The file gateway gradually receives the file data sent by the client and stores the files in the shard cache area obtained according to the pre-sharding. The specific example is shown in the figure Figure 3As shown, the shard cache area includes shard hot areas and non-shard hot areas. The shard hot areas are dark-marked areas. The cached file data will be stored in the cache space until the file transfer is completed. For example, Figure 3 After A is uploaded to the object storage server, it still exists in the cache space; the non-shard hot area is the white area, and the file data in it will be overwritten by subsequent file data after being uploaded to the object storage server. For example, Figure 3 The BC in is overwritten by the DE after being uploaded to the object storage server.

[0088] The file data cached in the cache space is uploaded to the object storage server. When the file data is subsequently received, the file data in the previous non-shard hot zone is overwritten by the subsequently received file data, and the overwritten file data is uploaded to the object storage server. For the file data in the shard hot zone, no overwriting operation is performed.

[0089] Before the object storage server has received all the file data, when the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is directly modified accordingly, and the modified file data cached in the shard hot zone is uploaded to the object storage server. The specific example is shown in the figure Figure 4 As shown, A is modified to G and uploaded to the object storage server. After all file data modifications are completed and the object storage server receives all file data, the object storage server is instructed to merge all file data to complete the file upload.

[0090] This application implements parallel uploading of shards by calculating offsets in pre-sharding, which can speed up the upload of large files, thereby improving the service quality and user experience of cloud storage.

[0091] This application relies on the mechanism of setting up hot zone cache data, reducing the large amount of upload overhead that may be caused by modification requests and improving the performance of file writing. For uploading files that require frequent access to hot data, the performance improvement is more significant.

[0092] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0093] Based on the same inventive concept, the embodiment of the present application also provides a file uploading device for implementing the file uploading method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file uploading device embodiments provided below can refer to the limitations on the file uploading method above, and will not be repeated here.

[0094] In an exemplary embodiment, Figure 5 As shown, a file uploading device is provided, including: a segmentation module 501, a receiving module 502, a storage module 503 and an uploading module 504, wherein:

[0095] The sharding module is used to pre-shard the local cache space based on the file sharding rules of object storage.

[0096] The receiving module is used to receive the file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area.

[0097] The storage module is used to upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone.

[0098] The upload module is used to modify the file data cached in the shard hot zone when it is necessary to modify the file data cached in the shard hot zone during the file data transmission process, and upload the file data cached in the shard hot zone to the object storage server to instruct the object storage server to merge based on the received file data.

[0099] In some embodiments, the above-mentioned upload module is also used to fill the corresponding upload bits of other file data based on preset characters during the process of file data transmission, and upload the filled results together with the file data cached in the target non-sharded hot zone to the object storage server together.

[0100] In some embodiments, the above-mentioned upload module is also used to download the file data cached by the target non-sharded hot zone from the object storage server, modify it, and upload the modified file data to the object storage server when the file data cached by the target non-sharded hot zone needs to be modified.

[0101] In some embodiments, the device further includes a partitioning module for determining the size of target file data to be stored in the shard hot zone and the offset in the file according to the historical modification frequency of the file data in the file to be stored as an object;

[0102] According to the size of the target file data and the offset of the target file data in the file, the cache space is divided into shard hot zones.

[0103] In some embodiments, in some embodiments, the above-mentioned upload module is also used to upload the file data cached in the cache space to the object storage server when the cache space is full.

[0104] In some embodiments, the apparatus further comprises a determination module, for determining the size and position number of the target file data in the shard cache area according to the shard cache area obtained by pre-sharding;

[0105] Subtract one from the position number and multiply it by the size of the target file data to obtain the offset of the target file data.

[0106] Each module in the above-mentioned file uploading device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module above.

[0107] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file upload method is implemented.

[0108] The display unit of the computer device is used to form a visually visible picture, and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball, or touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse, etc.

[0109] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0111] Pre-shard the local cache space based on the file sharding rules of object storage;

[0112] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0113] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0114] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0115] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0116] Pre-shard the local cache space based on the file sharding rules of object storage;

[0117] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0118] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0119] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0120] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0121] Pre-shard the local cache space based on the file sharding rules of object storage;

[0122] Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0123] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone;

[0124] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0126] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0128] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file uploading method, characterized in that: The method comprises: Pre-shard the local cache space based on the file sharding rules of object storage; Receive file data sent by a client, and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area; Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the file data received subsequently needs to be overwritten and stored in the non-shard hot zone; During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

2. The method according to claim 1, characterized in that The method further comprises: During the process of file data transmission, if there is file data cached in the target non-shard hot zone that needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, the corresponding upload bits of the other file data are filled based on preset characters, and the filled results are uploaded to the object storage server together with the file data cached in the target non-shard hot zone.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the file data cached in the target non-shard hot zone needs to be modified, the file data cached in the target non-shard hot zone is downloaded from the object storage server, modified, and the modified file data is uploaded to the object storage server.

4. The method according to claim 1, characterized in that: The division method of the shard hot zone includes: According to the historical modification frequency of the file data in the file to be stored in the object, determine the size of the target file data to be stored in the shard hot zone and the offset in the file; According to the size of the target file data and the offset of the target file data in the file, the cache space is divided into shard hot zones.

5. The method according to claim 1, characterized in that The step of uploading the file data cached in the cache space to the object storage server includes: When the cache space is full, the file data cached in the cache space is uploaded to the object storage server.

6. The method according to claim 4, characterized in that Determining the size of the target file data to be stored in the shard hot zone and the offset in the file includes: Determine the size and position number of the target file data in the shard cache area according to the shard cache area obtained by pre-sharding; The position number is subtracted by one, and then multiplied by the size of the target file data to obtain the offset of the target file data.

7. A file uploading device, characterized in that: The device comprises: The sharding module is used to pre-shard the local cache space based on the file sharding rules of object storage; A receiving module, used for receiving file data sent by a client, and caching the file data according to a shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area; A storage module is used to upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone; The upload module is used to modify the file data cached in the shard hot zone when it is necessary to modify the file data cached in the shard hot zone during the file data transmission process, and upload the file data cached in the shard hot zone to the object storage server to instruct the object storage server to merge based on the received file data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images