File uploading method, device, computer equipment, readable storage medium and program product

Abstract

The present application relates to a file uploading method, apparatus, computer equipment, computer-readable storage medium and computer program product. The method comprises: pre-sharding the local cache space based on the file sharding rules of object storage; receiving the file data sent by the client, and caching the file data according to the sharding cache area obtained by pre-sharding; uploading the file data cached in the cache space to the object storage server, retaining the file data cached in the sharding hot zone, and overwriting and storing the file data received subsequently when it needs to be overwritten and stored in the non-sharding hot zone; when it is necessary to modify the file data cached in the sharding hot zone, modify the file data cached in the sharding hot zone, and upload the file data cached in the sharding hot zone to the object storage server to instruct the object storage server to merge based on the received file data. The use of this method can effectively improve the efficiency of file uploading.

Description

Technical Field

[0001] The present application relates to the field of file storage technology, and in particular to a file uploading method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous advancement of modern computer network technology, object-based file gateways have become an important data storage method. By providing a standard file system interface, they enable users to easily interact with object storage. In object-based file gateway applications, uploading large files inherently presents a significant time and cost challenge. Further complicating matters, when large files require simultaneous read-modify-write (RMW) operations during upload, this double burden significantly increases processing time and resource consumption.

[0003] In related technologies, when an RMW request occurs during a multi-segment upload, if the requested portion doesn't exist in the cache, all consecutive segments in the cache must first be uploaded and then completed. The portion to be modified is then read into the cache for modification, and a new segment is created and uploaded before the RMW request is completed. This continuous creation of multi-segment upload operations and the execution of numerous uploads results in a large number of repeated object data operations, resulting in slow file upload speeds. Summary of the Invention

[0004] Based on this, it is necessary to provide a file uploading method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the file uploading rate in order to address the above technical problems.

[0005] In a first aspect, the present application provides a method for uploading a file, comprising:

[0006] Pre-shard the local cache space based on the file sharding rules of object storage;

[0007] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0008] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0009] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0010] In one embodiment, the method further comprises:

[0011] During the process of file data transmission, if the file data cached in the target non-sharded hot zone needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, the corresponding upload bits of the other file data are filled based on the preset characters, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

[0012] In one embodiment, the method further comprises:

[0013] When the file data cached in the target non-sharded hot zone needs to be modified, the file data cached in the target non-sharded hot zone is downloaded from the object storage server, modified, and the modified file data is uploaded to the object storage server.

[0014] In one embodiment, the division of the shard hot zones includes:

[0015] Determine the size of the target file data to be stored in the shard hot zone and its offset in the file based on the historical modification frequency of the file data in the file to be stored in the object;

[0016] According to the size of the target file data and the offset of the target file data in the file, the cache space is divided into shard hot zones.

[0017] In one embodiment, uploading the file data cached in the cache space to the object storage server includes:

[0018] When the cache space is full, the file data cached in the cache space is uploaded to the object storage server.

[0019] In one embodiment, determining the size of target file data to be stored in the shard hot zone and the offset in the file includes:

[0020] Determine the size and location number of the target file data in the shard cache area according to the shard cache area obtained by pre-sharding;

[0021] The position number is subtracted by one, and the result is multiplied by the size of the target file data to obtain the offset of the target file data.

[0022] In a second aspect, the present application further provides a file uploading device, comprising:

[0023] The sharding module is used to pre-shard the local cache space based on the file sharding rules of object storage;

[0024] A receiving module, configured to receive file data sent by a client and cache the file data according to a shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area;

[0025] A storage module is used to upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone;

[0026] The upload module is used to modify the file data cached in the shard hot zone when it is necessary to modify the file data cached in the shard hot zone during the file data transmission process, and upload the file data cached in the shard hot zone to the object storage server to instruct the object storage server to merge based on the received file data.

[0027] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0028] Pre-shard the local cache space based on the file sharding rules of object storage;

[0029] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0030] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0031] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Pre-shard the local cache space based on the file sharding rules of object storage;

[0034] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0035] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0036] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0037] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0038] Pre-shard the local cache space based on the file sharding rules of object storage;

[0039] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0040] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0041] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0042] The above-mentioned file uploading method, device, computer equipment, computer-readable storage medium and computer program product first pre-shard the local cache space based on the file sharding rules of object storage; receive the file data sent by the client, and cache the file data according to the sharding cache area obtained by pre-sharding; upload the file data cached in the cache space to the object storage server, retain the file data cached in the sharding hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-sharding hot zone; during the file data transmission process, if the file data cached in the sharding hot zone needs to be modified, the file data cached in the sharding hot zone is modified until the file data cached in the sharding hot zone no longer needs to be modified, and the file data cached in the sharding hot zone is uploaded to the object storage server to instruct the object storage server to merge the received file data. In this way, by pre-sharding and setting the sharding hot zone, the large amount of upload overhead caused by the need to modify the file is reduced, and the file upload efficiency is accelerated. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 This is an application environment diagram of a file uploading method in one embodiment;

[0045] Figure 2 Schematic diagram of a process for uploading a file in one embodiment;

[0046] Figure 3 A schematic diagram of an object storage server for uploading file data in one embodiment;

[0047] Figure 4 A schematic diagram of modifying and uploading file data to an object storage server in one embodiment;

[0048] Figure 5 This is a structural block diagram of a file uploading device in one embodiment;

[0049] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] The file upload method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the file gateway 102 communicates with the object storage server 104 via a network. The data storage system can store data that the object storage server 104 needs to process. The data storage system can be integrated with the object storage server 104 or placed on the cloud or other network servers.

[0052] The client determines the file data that needs to be uploaded and writes the file data into the cache space in the file gateway 102. After receiving the file data uploaded by the client, the file gateway 102 uploads the file data to the object storage server. After receiving all the file data, the object storage server merges the received file data to obtain the file data that needs to be uploaded.

[0053] Clients include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices include smart speakers, smart TVs, smart air conditioners, smart car devices, and projectors. Portable wearable devices include smart watches, smart bracelets, and head-mounted devices. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, and the like. Object storage server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a file upload method is provided, which is applied to Figure 1 The file gateway 102 in FIG. 1 is used as an example to illustrate the process, including the following steps 202 to 208. Among them:

[0055] Step 202: pre-shard the local cache space based on the file sharding rule of the object storage.

[0056] Among them, object storage is a data storage architecture that stores data in the form of objects in a storage system.

[0057] For example, the file gateway pre-shards the local cache space according to the file sharding rules of the object storage. The size and location of the pre-shards can be set according to the type of specific file data, and this embodiment of the application does not limit this.

[0058] Step 204: Receive the file data sent by the client, and cache the file data according to the fragment cache area obtained by pre-fragmentation.

[0059] The shard cache area includes a shard hot zone and a non-shard hot zone. The file data may be a video package file, an image file, or a word document, or may be file data in other formats, which is not limited in the present embodiment.

[0060] Optionally, the file gateway gradually receives file data sent by the client and stores the files in the shard cache area obtained according to pre-sharding. The shard cache area includes shard hot areas and non-shard hot areas. The file data cached in the shard hot areas will be stored in the cache space until the end of the file transfer; the file data in the non-shard hot areas will be overwritten by subsequent file data after being uploaded to the object storage server.

[0061] Step 206: Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone.

[0062] Exemplarily, the file data cached in the cache space is uploaded to the object storage server. When the file data is subsequently received, the subsequently received file data overwrites the file data in the previous non-sharded hot zone, and the overwritten file data is uploaded to the object storage server. For the file data in the sharded hot zone, no overwriting operation is performed.

[0063] Step 208: During the file data transmission process, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified, and the file data cached in the shard hot zone is uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0064] Optionally, before the object storage server has received all the file data, when it is necessary to modify the file data cached in the shard hot zone, the file data cached in the shard hot zone is directly modified accordingly, and the modified file data cached in the shard hot zone is uploaded to the object storage server. After all modifications to the file data are completed and the object storage server has received all the file data, the object storage server is instructed to merge all the file data to complete the file upload.

[0065] In some embodiments, during the file transfer process, the file data cached in the shard hot zone may be modified multiple times. After the multiple modifications are completed and the object storage server receives all the file data, the object storage server is instructed to merge all the file data to complete the file upload.

[0066] In the above-mentioned file upload method, the local cache space is pre-sharded based on the file sharding rules of the object storage; the file data sent by the client is received, and the file data is cached according to the sharding cache area obtained by pre-sharding; the file data cached in the cache space is uploaded to the object storage server, and the file data cached in the sharding hot zone is retained. When the file data received subsequently needs to be overwritten and stored in the non-sharding hot zone, the file data is overwritten and stored; during the file data transmission process, when the file data cached in the sharding hot zone needs to be modified, the file data cached in the sharding hot zone is modified until the file data cached in the sharding hot zone no longer needs to be modified, and the file data cached in the sharding hot zone is uploaded to the object storage server to instruct the object storage server to merge based on the received file data. In this way, by pre-sharding and setting sharding hot zones, the large amount of upload overhead caused by the need to modify files is reduced, and the file upload efficiency is accelerated.

[0067] In an exemplary embodiment, the file upload method also includes: during the file data transmission process, if there is file data cached in the target non-sharded hot zone that needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, the corresponding upload bits of the other file data are filled based on preset characters, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

[0068] In actual implementation, during the process of file data transmission, when the file data cached in the target non-sharded hot zone needs to be uploaded to the object server together, but other file data is not cached in the cache space, the preset characters are used to fill in the corresponding upload bits of other file data, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

[0069] For example, during the file transfer process, ABCDE file data needs to be uploaded to the object storage server, but only AD data is cached in the cache space, with a total of 5 upload bits. At this time, the remaining upload bits can be filled with 0 and uploaded together, that is, A00D0 file data is uploaded to the object storage server.

[0070] In the above embodiment, when uploading is required, preset characters are used to fill the upload position that is not cached in the cache space and then the upload is performed, thereby avoiding the situation where the file data is overwritten due to untimely uploading, which ultimately leads to the loss of file data.

[0071] In an exemplary embodiment, the file upload method also includes: when it is necessary to modify the file data cached in the target non-sharded hot zone, downloading the file data cached in the target non-sharded hot zone from the object storage server, modifying it, and uploading the modified file data to the object storage server.

[0072] In actual implementation, when the file data in the cache space needs to be modified and the modified file data is located in the target non-sharded hot zone, the file data cached in the target non-sharded hot zone is downloaded from the object storage server and modified, and the modified file data is uploaded to the object storage server.

[0073] For example, when A in the target non-shard hot zone needs to be modified, the A00D0 file data is downloaded from the object storage server, A is modified to obtain A', and then the A'00D0 file data is uploaded to the object storage server.

[0074] In the above embodiment, by downloading the corresponding file data and modifying it, and then re-uploading it after the modification, a faster file data modification operation is achieved, thereby improving the file upload rate.

[0075] In an exemplary embodiment, the method of dividing the shard hot zone includes: determining the size of the target file data to be stored in the shard hot zone and the offset in the file based on the historical modification frequency of the file data in the file to be object stored; and dividing the shard hot zone in the cache space based on the size of the target file data and the offset of the target file data in the file.

[0076] In actual implementation, the format and usage scenario of the file data in the file to be object stored are obtained, and the size, offset and modification frequency of each file data in the file data are determined based on the format and usage scenario of the file data. When the modification frequency of a certain file data exceeds the preset number of times, the cache space is divided into shard hot zones based on the corresponding size and offset of the corresponding file data.

[0077] In the above embodiment, by presetting the shard hot zone, when the file data needs to be modified, the corresponding modification can be directly performed and then uploaded, which reduces the overhead required for the modification process and improves the performance of file upload.

[0078] In an exemplary embodiment, uploading the file data cached in the cache space to the object storage server includes: when the cache space is full, uploading the file data cached in the cache space to the object storage server.

[0079] In actual implementation, when the cache space is full, the file data cached in the cache space is uploaded to the object storage server.

[0080] In another embodiment, when the file data in the cache space meets a preset condition, the file data cached in the cache space is uploaded to the object storage server. The preset condition may be uploading the file data cached in the cache space to the object storage server once every 30 seconds, or other conditions may be set by the user based on actual circumstances, and this application does not limit this.

[0081] In the above embodiment, when the cache space is full, the file data in the cache space is uploaded to the object storage server, thereby preventing the possibility that the subsequently uploaded file data will overwrite the existing file data.

[0082] In an exemplary embodiment, determining the size of target file data to be stored in a shard hot zone and its offset in the file includes: determining the size and position number of the target file data in the shard cache area based on the shard cache area obtained by pre-sharding; subtracting one from the position number and multiplying it by the size of the target file data to obtain the offset of the target file data.

[0083] In actual implementation, the size and position number of the target file data in the shard cache area are determined based on the shard cache area obtained by pre-sharding and combined with the target file data; the position number is subtracted by one and then multiplied by the size of the target file data to obtain the offset of the target file data.

[0084] In the above embodiment, by determining the offset of the file data, concurrent uploading according to the offset can be achieved, thereby accelerating the upload rate of the file data.

[0085] To explain the file upload method in this application in detail, an embodiment is used below for illustration. For example, this application describes the file upload method in a specific file upload scenario.

[0086] First, the file gateway pre-shards the local cache space according to the file sharding rules of the object storage.

[0087] The file gateway gradually receives the file data sent by the client and stores the files in the shard cache area obtained according to the pre-sharding. The specific example is shown in the figure Figure 3As shown, the shard cache area includes shard hot areas and non-shard hot areas. The shard hot areas are dark-marked areas. The cached file data will be stored in the cache space until the file transfer is completed. For example, Figure 3 After being uploaded to the object storage server, A in the figure still exists in the cache space. The non-sharded hot area is the white area, where the file data will be overwritten by subsequent file data after being uploaded to the object storage server. For example, Figure 3 The BC in the file is overwritten by the DE after being uploaded to the object storage server.

[0088] The file data cached in the cache space is uploaded to the object storage server. When the file data is subsequently received, the file data in the non-sharded hot zone is overwritten by the subsequent file data, and the overwritten file data is uploaded to the object storage server. For the file data in the sharded hot zone, no overwriting operation is performed.

[0089] Before the object storage server has received all the file data, when the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified directly, and the modified file data cached in the shard hot zone is uploaded to the object storage server. The specific example is shown in the figure Figure 4 As shown, A is modified to G and uploaded to the object storage server. After all file data modifications are completed and the object storage server receives all file data, the object storage server is instructed to merge all file data to complete the file upload.

[0090] This application implements parallel uploading of shards by calculating offsets based on pre-sharding, which can speed up the upload of large files, thereby improving the service quality and user experience of cloud storage.

[0091] This application relies on a mechanism to cache data in hot zones, reducing the large upload overhead that may be caused by modification requests and improving file write performance. The performance improvement is even more significant in scenarios where uploading files requires frequent access to hot data.

[0092] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0093] Based on the same inventive concept, embodiments of the present application also provide a file upload device for implementing the aforementioned file upload method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more file upload device embodiments provided below can be found in the above-mentioned limitations of the file upload method and will not be repeated here.

[0094] In an exemplary embodiment, Figure 5 As shown, a file uploading device is provided, comprising: a fragmentation module 501, a receiving module 502, a storage module 503 and an uploading module 504, wherein:

[0095] The sharding module is used to pre-shard the local cache space based on the file sharding rules of object storage.

[0096] The receiving module is used to receive file data sent by the client and cache the file data according to the fragment cache area obtained by pre-fragmentation; wherein the fragment cache area includes a fragment hot area and a non-fragment hot area.

[0097] The storage module is used to upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone.

[0098] The upload module is used to modify the file data cached in the shard hot zone when it is necessary to modify the file data cached in the shard hot zone during the file data transmission process, and upload the file data cached in the shard hot zone to the object storage server to instruct the object storage server to merge based on the received file data.

[0099] In some embodiments, the above-mentioned upload module is also used to fill the corresponding upload bits of other file data based on preset characters during the process of file data transmission, when there is file data cached in the target non-sharded hot zone that needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

[0100] In some embodiments, the above-mentioned upload module is also used to download the file data cached by the target non-sharded hot zone from the object storage server, modify it, and upload the modified file data to the object storage server when the file data cached by the target non-sharded hot zone needs to be modified.

[0101] In some embodiments, the apparatus further includes a partitioning module for determining the size of target file data to be stored in the shard hot zone and the offset in the file based on the historical modification frequency of the file data in the file to be stored as the object;

[0102] According to the size of the target file data and the offset of the target file data in the file, the cache space is divided into shard hot zones.

[0103] In some embodiments, the upload module is further configured to upload the file data cached in the cache space to the object storage server when the cache space is full.

[0104] In some embodiments, the apparatus further comprises a determination module for determining the size and position number of the target file data in the shard cache area based on the shard cache area obtained by pre-sharding;

[0105] Subtract one from the position number and multiply it by the size of the target file data to obtain the offset of the target file data.

[0106] Each module in the above-mentioned file upload device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0107] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface, a display unit and an input device. The processor, memory and input / output interface are connected via a system bus, and the communication interface, display unit and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store file data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a file uploading method is implemented.

[0108] The display unit of the computer device is used to produce a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, a keypad, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0109] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0110] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0111] Pre-shard the local cache space based on the file sharding rules of object storage;

[0112] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0113] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0114] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Pre-shard the local cache space based on the file sharding rules of object storage;

[0117] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0118] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0119] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0120] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0121] Pre-shard the local cache space based on the file sharding rules of object storage;

[0122] Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone;

[0123] Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone;

[0124] During the process of file data transmission, if the file data cached in the shard hot zone needs to be modified, the file data cached in the shard hot zone is modified and uploaded to the object storage server to instruct the object storage server to merge based on the received file data.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0126] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0127] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0128] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A file uploading method, characterized in that: The method comprises: Pre-shard the local cache space based on the file sharding rules of object storage; Receive file data sent by the client and cache the file data according to the shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot zone and a non-shard hot zone; Upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and overwrite and store the file data received subsequently when it needs to be overwritten and stored in the non-shard hot zone; During the file data transmission process, if the file data cached in the shard hot zone needs to be modified multiple times, the file data cached in the shard hot zone is modified, and the file data cached in the shard hot zone is uploaded to the object storage server to instruct the object storage server to merge based on the received file data; During the process of file data transmission, if the file data cached in the target non-sharded hot zone needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space, the corresponding upload bits of the other file data are filled based on the preset characters, and the filled results are uploaded to the object storage server together with the file data cached in the target non-sharded hot zone.

2. The method according to claim 1, characterized in that The method further comprises: When the file data cached in the target non-sharded hot zone needs to be modified, the file data cached in the target non-sharded hot zone is downloaded from the object storage server, modified, and the modified file data is uploaded to the object storage server.

3. The method according to claim 1, characterized in that The division method of the shard hot zone includes: Determine the size of the target file data to be stored in the shard hot zone and its offset in the file based on the historical modification frequency of the file data in the file to be stored in the object; According to the size of the target file data and the offset of the target file data in the file, the cache space is divided into shard hot zones.

4. The method according to claim 1, wherein The uploading of the file data cached in the cache space to the object storage server includes: When the cache space is full, the file data cached in the cache space is uploaded to the object storage server.

5. The method according to claim 3, characterized in that Determining the size of the target file data to be stored in the shard hot zone and the offset in the file includes: Determine the size and location number of the target file data in the shard cache area according to the shard cache area obtained by pre-sharding; The position number is subtracted by one, and the result is multiplied by the size of the target file data to obtain the offset of the target file data.

6. A file uploading device, characterized in that: The device comprises: The sharding module is used to pre-shard the local cache space based on the file sharding rules of object storage; A receiving module, configured to receive file data sent by a client and cache the file data according to a shard cache area obtained by pre-sharding; wherein the shard cache area includes a shard hot area and a non-shard hot area; A storage module is used to upload the file data cached in the cache space to the object storage server, retain the file data cached in the shard hot zone, and perform overwriting storage when the subsequently received file data needs to be overwritten and stored in the non-shard hot zone; an upload module, configured to, during the process of file data transmission, modify the file data cached in the shard hot zone if the file data cached in the shard hot zone needs to be modified, and upload the file data cached in the shard hot zone to the object storage server, so as to instruct the object storage server to merge the file data based on the received file data; The upload module is also used to fill the corresponding upload bits of other file data based on preset characters during the process of file data transmission, and upload the filled results together with the file data cached in the target non-shard hot zone to the object storage server, if the file data cached in the target non-shard hot zone needs to be uploaded to the object storage server together with other file data, and the other file data is not cached in the cache space.

7. The device according to claim 6, characterized in that The upload module is also used for: When the file data cached in the target non-sharded hot zone needs to be modified, the file data cached in the target non-sharded hot zone is downloaded from the object storage server, modified, and the modified file data is uploaded to the object storage server.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images