File storage processing method and device, storage platform, medium and product

Abstract

The application provides a file storage processing method and device, a storage platform, a medium and a product. It relates to the field of data processing. The storage platform comprises a server, a tracking server and a storage server. The method comprises the following steps: the server receives an upload request for a to-be-processed file sent by a client, and sends the upload request to the tracking server if the hash value of the to-be-processed file does not exist in the database; the tracking server obtains the IP address and port number of the storage server and sends them to the server; the server sends the to-be-processed file to the storage server for storage according to the IP address and the port number; if the storage parameter is less than the threshold value of combined storage, the to-be-processed file is stored in an idle storage block; if the storage parameter is greater than the threshold value of fragmented storage and a plurality of fragmented files sent by the server are received, the plurality of fragmented files are stored in sequence. The method of the application avoids wasting storage space, improves transmission efficiency and reduces transmission time.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, storage platform, medium and product for file storage. Background Technology

[0002] With the development of information technology, the total amount of data generated by various industries has exploded. When faced with files of varying sizes, how to adopt appropriate storage strategies to store files has become an urgent problem to be solved.

[0003] Currently, existing technologies typically employ traditional single-machine storage methods to directly store files of varying sizes.

[0004] However, this existing method does not differentiate between file sizes. On the one hand, it wastes storage space when storing small files; on the other hand, it reduces transmission efficiency and increases transmission time when storing large files. Summary of the Invention

[0005] This application provides a file storage processing method, apparatus, storage platform, medium, and product to solve the problems of wasted storage space, reduced transmission efficiency during storage, and increased transmission time in the prior art.

[0006] Firstly, this application provides a file storage processing method, comprising: a server receiving an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed; the server determining whether a hash value exists in a database; if the server determines that a hash value does not exist in the database, the server sending an upload request to a tracking server; the tracking server obtaining the IP address and port number of the storage server based on the upload request, and sending the IP address and port number to the server; the server sending the file to be processed to the storage server for storage based on the IP address and port number; the storage server performing the following steps on the file to be processed during the storage process: if the storage parameters are determined to be less than the required hash value... If a storage threshold is set, a free storage block identifier is obtained through the free balance tree, and the file to be processed is stored in the free storage block corresponding to the free storage block identifier. The free storage block is used to merge and store files to be processed whose storage parameters are less than the merge storage threshold. If it is determined that the storage parameters are greater than the fragment storage threshold, and multiple fragment files are received from the server, the multiple fragment files are stored sequentially. The multiple fragment files are obtained by the client fragmenting the file to be processed and sending the fragmented file to be processed to the server. After the storage server completes the storage of the file to be processed, it sends the file identifier of the file to be processed to the client, whereby the file identifier identifies the file storage path of the file to be processed.

[0007] In one possible design, obtaining the free storage block identifier through the free balance tree includes: determining whether there is a free storage block in the free balance tree; if it is determined that there is a free storage block in the free balance tree, obtaining the free storage block identifier of the free balance block; if it is determined that there is no free storage block in the free balance tree, adding a new free storage block in the free balance tree and obtaining the free storage block identifier of the new free storage block.

[0008] In one possible design, each of the multiple fragment files carries a fragment number; correspondingly, the multiple fragment files are stored sequentially, including: storing the first fragment file in the multiple fragment files according to the fragment number, and obtaining the file identifier and the offset of the first fragment file; for each of the remaining fragment files, the following steps are performed sequentially according to the fragment number to complete the storage: storing the current fragment file according to the file identifier and the offset of the previous fragment file, and obtaining the offset of the current fragment file.

[0009] In one possible design, before storing the current segment file, the following steps are included: the server verifies the current segment file, and if the verification passes, the current segment file is sent to the storage server for storage; correspondingly, after obtaining the offset of the current segment file, the following steps are included: the storage server sends the file identifier and the offset of the current segment file to the server; the server saves the file's hash value, file identifier, the offset of the current segment file, and the segment number of each segment file to the cache.

[0010] In one possible design, the server verifies the current fragment file. If the verification passes, the server sends the current fragment file to the storage server for storage. This includes: the server obtaining the maximum fragment number corresponding to the hash value in the cache; if the maximum fragment number is less than the fragment number of the current fragment file, the verification passes, and the current fragment file is sent to the storage server for storage.

[0011] In one possible design, after the server checks whether a hash value exists in the database, it also includes: if the server determines that a hash value exists in the database, it obtains the file identifier of the file; the server sends the file identifier to the client.

[0012] Secondly, this application provides a file storage processing apparatus, comprising:

[0013] The upload module is used by the server to receive upload requests from the client for files to be processed. The upload request carries the hash value and storage parameters of the file to be processed.

[0014] The judgment module is used by the server to determine whether a hash value exists in the database;

[0015] The first sending module is used to send an upload request to the tracking server if the server determines that the hash value does not exist in the database.

[0016] The second sending module is used to track the server to obtain the IP address and port number of the storage server based on the upload request, and send the IP address and port number to the server.

[0017] The storage module is used by the server to send the files to be processed to the storage server for storage based on the IP address and port number; the storage server performs the following steps on the files to be processed during the storage process;

[0018] The storage module includes:

[0019] The merge storage unit is used to obtain the free storage block identifier through the free balance tree if the storage parameter is determined to be less than the merge storage threshold, and to store the file to be processed into the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the files to be processed whose storage parameter is less than the merge storage threshold.

[0020] The fragmented storage unit is used to store multiple fragmented files sequentially if the storage parameters are determined to be greater than the fragmented storage threshold and multiple fragmented files are received from the server. The multiple fragmented files are obtained by the client fragmenting the file to be processed and sending the fragmented file to be processed to the server.

[0021] The third sending module is used to send the file identifier of the file to be processed to the client after the storage server has finished storing the file to be processed. The file identifier identifies the file storage path of the file to be processed.

[0022] Thirdly, embodiments of this application provide a storage platform, including: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the file storage processing method as described in the first aspect and various possible designs of the first aspect.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the file storage processing method described in the first aspect and various possible designs of the first aspect.

[0024] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the file storage processing method described in the first aspect and various possible designs of the first aspect.

[0025] The file storage processing method, apparatus, storage platform, medium, and product provided in this application, if the file's storage parameters are less than the merged storage threshold, it indicates that the file is a small file. The file is then stored in a free storage block after the storage unit has been divided, and subsequently merged with files in other free storage blocks, avoiding the waste of storage space caused by storing the file independently in a single storage unit. If the storage parameters are greater than the fragmented storage threshold, it indicates that the file is a large file. The file is fragmented into multiple fragments for storage, which can improve the transmission efficiency during the storage process and reduce transmission time. Furthermore, by determining the hash value of the file to be processed, duplicate files to be processed can be avoided from being stored on the storage server. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] Figure 1 An application scenario diagram illustrating the file storage processing method provided in the embodiments of this application;

[0028] Figure 2 A flowchart illustrating the file storage processing method provided in this application embodiment;

[0029] Figure 3 A schematic diagram of the structure of a file storage processing apparatus provided in an embodiment of this application;

[0030] Figure 4 A schematic diagram of the structure of the storage platform provided in the embodiments of this application.

[0031] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0034] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0035] With the development of information technology, the total amount of data generated by various industries has exploded. How to adopt appropriate storage strategies for files of varying sizes has become an urgent problem to be solved. Currently, existing technologies typically use traditional single-machine storage methods to directly store files of different sizes. However, this method does not differentiate between file sizes. On the one hand, it wastes storage space when storing smaller files; on the other hand, it reduces transmission efficiency and increases transmission time when storing larger files.

[0036] To address the aforementioned technical problems, this application proposes the following technical concept: Considering that failing to differentiate file data sizes wastes storage space and reduces transmission efficiency and increases transmission time during storage, the inventors devised a method to differentiate files based on data size. Smaller files are merged for storage, while larger files are fragmented for storage. If a file's storage parameters are less than the merging threshold, it indicates a smaller file, which is then stored in a free storage block after the storage unit has been divided. Subsequently, it is merged with files in other free storage blocks, avoiding the waste of storage space by storing the file independently in a single storage unit. If the storage parameters are greater than the fragmentation threshold, it indicates a larger file, which is then fragmented into multiple fragments for storage, improving transmission efficiency and reducing transmission time during storage.

[0037] Figure 1 This diagram illustrates an application scenario for the file storage processing method provided in this embodiment. For example... Figure 1 As shown, the application scenario includes: client 10 and storage platform 20.

[0038] The storage platform 20 includes: server 201, tracking server 202 and storage server 203.

[0039] refer to Figure 1Client 10 sends upload requests for files to be processed to server 201. Storage server 203 periodically sends information about itself, including its IP address and port number, to tracking server 202. Tracking server 202 sends its IP address and port number to server 201. Server 201 sends files to be processed to storage server 203 for storage based on the IP address and port number.

[0040] Figure 2 This is a flowchart of a file storage processing method provided in an embodiment of this application; the execution entity of this embodiment can be... Figure 1 The storage platform in the illustrated embodiment includes a server, a tracking server, and a storage server. Other devices with similar functions may also be included; this embodiment does not impose any particular limitations. Figure 2 As shown, the method includes:

[0041] S201: The server receives an upload request from the client for a file to be processed, in which the upload request carries the hash value and storage parameters of the file to be processed.

[0042] The storage parameter indicates the size of the file to be processed.

[0043] In this embodiment, the hash value of the file to be processed is calculated by the client using a hash algorithm.

[0044] Optionally, the hash algorithm can be Message-Digest Algorithm 5 (MD5). MD5 is a type of hash algorithm, and the hash value can be an MD5 value.

[0045] S202: The server checks whether a hash value exists in the database.

[0046] S203: If the server determines that the hash value does not exist in the database, the server sends an upload request to the tracking server.

[0047] In this embodiment, if the hash value does not exist in the database, it means that the file to be processed corresponding to the hash value has not been uploaded.

[0048] Optionally, if the server determines that a hash value exists in the database, it obtains the file identifier of the file; the server then sends the file identifier to the client.

[0049] In this embodiment, if a hash value exists in the database, it means that the file to be processed has been stored in the storage server. The server then sends a file identifier request to the tracking server, which sends the file identifier back to the server. The server then sends the file identifier to the client, and the client notifies the user that the upload was successful.

[0050] S204: The tracking server obtains the IP address and port number of the storage server based on the upload request, and sends the IP address and port number to the server.

[0051] Optionally, the storage server periodically sends the storage server's operational status and some important information to the tracking server, such as the number of uploaded files, the number of deleted files, the total number of downloads, the remaining memory space, and file synchronization information.

[0052] In this embodiment, the tracking server obtains the available IP address and port number from the storage server based on the upload request.

[0053] S205: The server sends the file to be processed to the storage server for storage based on the IP address and port number; the storage server performs the following steps on the file to be processed during the storage process:

[0054] In this embodiment, the server communicates with the storage server based on the IP address and port number, and sends the file to be processed to the storage server for storage.

[0055] Specifically, step S205 includes S2051~S2052:

[0056] S2051: If the storage parameters are determined to be less than the merge storage threshold, the free storage block identifier is obtained through the free balance tree, and the file to be processed is stored in the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the files to be processed whose storage parameters are less than the merge storage threshold.

[0057] Specifically, the storage space in the storage server is divided into multiple trunk blocks, each of which can be 64MB in size. Each trunk block is then further divided into multiple free storage blocks, each of which can be 16MB in size.

[0058] Optionally, the storage merge threshold is 16MB. When the storage parameter is less than 16MB, the files to be processed will be stored in a free storage block and merged with other files to be processed.

[0059] In this embodiment, after storing the file to be processed into the corresponding free storage block, the storage server stores the file identifier, Trunk ID, offset of the file to be processed in the Trunk, and merge storage time, etc. The file identifier identifies the file storage path of the file to be processed on the storage server.

[0060] Specifically, the process of obtaining free storage block identifiers through the free balance tree will be described in detail in subsequent embodiments.

[0061] S2052: If the storage parameter is determined to be greater than the fragment storage threshold, and multiple fragment files are received from the server, then the multiple fragment files are stored sequentially; the multiple fragment files are obtained by the client fragmenting the file to be processed and sending the fragmented file to be processed to the server.

[0062] Optionally, the fragment storage threshold can be 100MB.

[0063] In this embodiment, if the storage parameters of the file to be processed are greater than the fragment storage threshold, then the file to be processed needs to be fragmented for storage.

[0064] In this embodiment, when the client sends a file upload request, if the storage parameters are greater than the fragment storage threshold, the file to be processed is fragmented according to the fragment size, resulting in multiple fragment files. Each fragment file carries a fragment sequence number. The client sends the multiple fragment files to the server, and the server sends them to the storage server sequentially according to the fragment sequence number. The storage server then stores the multiple fragment files sequentially.

[0065] Specifically, based on the fragment number, the first fragment file among multiple fragment files is stored, and the file identifier and offset of the first fragment file are obtained; for each remaining fragment file, the following steps are performed sequentially according to the fragment number to complete the storage: based on the file identifier and the offset of the previous fragment file, the current fragment file is stored, and the offset of the current fragment file is obtained.

[0066] In this embodiment, based on the file identifier and the offset of the first fragment file, the second fragment file is stored according to the fragment number, and the offset of the second fragment file is obtained; based on the file identifier and the offset of the second fragment file, the third fragment file is stored according to the fragment number, and the offset of the third fragment file is obtained, and so on, until the offset of the second-to-last fragment file is obtained. Based on the file identifier and the offset of the second-to-last fragment file, the last fragment file is stored, thus completing the storage of multiple fragment files.

[0067] The offset is used to determine the location where each fragment file is stored.

[0068] Optionally, if the storage parameter is determined to be greater than the merged storage threshold but less than the fragmented storage threshold, the file to be processed is saved directly.

[0069] S206: After the storage server finishes storing the file to be processed, it sends the file identifier of the file to be processed to the client, where the file identifier identifies the file storage path of the file to be processed.

[0070] In this embodiment, after the storage server completes the storage of the file to be processed, it sends the file identifier of the file to be processed to the server. The server persists the file identifier and sends it to the client to notify the user that the upload was successful.

[0071] In summary, if the file's storage parameters are less than the merge storage threshold, the file is considered small. It should be stored in a free storage block after the storage unit has been divided, and subsequently merged with files in other free storage blocks. This avoids storing the file independently in a single storage unit, which would waste storage space. If the storage parameters are greater than the fragment storage threshold, the file is considered large. It should be fragmented into multiple fragments for storage, which improves transmission efficiency and reduces transmission time. Furthermore, by determining the hash value of the file to be processed, duplicate files can be avoided from being stored on the storage server.

[0072] Based on the above embodiments, this embodiment provides a detailed description of the process of obtaining free storage block identifiers through the free balanced tree, as detailed below:

[0073] S301: Determine if there are any free storage blocks in the free balanced tree.

[0074] In this embodiment, the storage server divides the storage space into multiple Trunk blocks, and each Trunk is further divided into multiple free storage blocks. Trunks use pre-allocated storage space to avoid duplicate allocation. To better manage free storage space, the storage server constructs a free balanced tree with free storage space as the key, and stores free storage blocks of the same size in a linked list. Optionally, the storage server is not part of a storage server cluster, and not all storage servers have the ability to manage and allocate free storage space. If the storage server receiving the file to be processed has nodes capable of managing and allocating free storage space, it determines whether there are free storage blocks in the free balanced tree.

[0075] Optionally, the criterion for determining whether there is a free storage block in the free balanced tree is: the free storage block is greater than and closest to the storage parameters of the file to be processed.

[0076] S302: If it is determined that there is a free storage block in the free balanced tree, then obtain the free storage block identifier of the free balanced block.

[0077] S303: If it is determined that there is no free storage block in the free balance tree, a new free storage block is added to the free balance tree, and the free storage block identifier of the new free storage block is obtained.

[0078] In this embodiment, if no free storage block exists, a new Trunk block is created, and the new Trunk block is divided into free storage blocks and added to the free balanced tree.

[0079] In summary, the idle balance tree can better manage the idle storage space of the storage server and further avoid wasting storage space.

[0080] Based on the above embodiments, this embodiment provides a detailed description of the processing before and after storing each file segment, as detailed below:

[0081] The server verifies the current fragment file. If the verification passes, the current fragment file is sent to the storage server for storage.

[0082] Specifically, the server retrieves the maximum shard number corresponding to the hash value in the cache. If the maximum shard number is less than the shard number of the current shard file, the verification passes, and the current shard file is sent to the storage server for storage.

[0083] In this embodiment, the cache stores information about the current file segment in key-value pairs. The key is a hash value, and the value includes the file identifier, offset, and maximum segment number. The server obtains the maximum segment number based on the hash value of the file to be processed. If the maximum segment number is less than the current file segment number, it means the current file segment has not been stored before, and the verification passes.

[0084] Optionally, if the maximum fragment sequence number is greater than the fragment sequence number of the current fragment file, it indicates that the current fragment file has already been stored, and the verification fails, skipping the current fragment file. This ensures the accuracy of fragment file uploads even in the event of network fluctuations or interruptions.

[0085] Optionally, after obtaining the offset of the current segment file, the storage server sends the file identifier and the offset of the current segment file to the server; the server saves the file hash value, file identifier, offset of the current segment file, and segment number of each segment file to the cache.

[0086] The offset is used to determine the location where each fragment file is stored.

[0087] In summary, by performing verification before storing the current file segment, we can avoid uploading duplicate file segments. Furthermore, if network fluctuations or interruptions occur, we can resume the transmission of the file segments based on the cache.

[0088] Figure 3 This is a schematic diagram of the structure of a file storage processing apparatus provided in an embodiment of this application. Figure 3As shown, the file storage processing device includes: an upload module 301, a judgment module 302, a first sending module 303, a second sending module 304, a storage module 305, and a third sending module 306. The storage module 305 includes: a merged storage unit 3051 and a fragmented storage unit 3052.

[0089] The upload module 301 is used for the server to receive upload requests for files to be processed sent by the client, wherein the upload request carries the hash value and storage parameters of the files to be processed;

[0090] Module 302 is used by the server to determine whether a hash value exists in the database;

[0091] The first sending module 303 is used to send an upload request to the tracking server if the server determines that the hash value does not exist in the database.

[0092] The second sending module 304 is used to track the server to obtain the IP address and port number of the storage server according to the upload request, and send the IP address and port number to the server.

[0093] Storage module 305 is used by the server to send the file to be processed to the storage server for storage based on the IP address and port number; the storage server performs the following steps on the file to be processed during the storage process;

[0094] The storage module 305 includes:

[0095] The merge storage unit 3051 is used to obtain the free storage block identifier through the free balance tree if the storage parameter is determined to be less than the merge storage threshold, and store the file to be processed into the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the files to be processed whose storage parameter is less than the merge storage threshold.

[0096] The fragmented storage unit 3052 is used to store multiple fragmented files sequentially if it is determined that the storage parameters are greater than the fragmented storage threshold and multiple fragmented files are received from the server; wherein the multiple fragmented files are obtained by the client fragmenting the file to be processed and sending the fragmented file to be processed to the server.

[0097] The third sending module 306 is used to send the file identifier of the file to be processed to the client after the storage server has completed storing the file to be processed. The file identifier identifies the file storage path of the file to be processed.

[0098] In one possible design, the merged storage unit 3051 is specifically used to: determine whether there is a free storage block in the free balance tree; if it is determined that there is a free storage block in the free balance tree, obtain the free storage block identifier of the free balance block; if it is determined that there is no free storage block in the free balance tree, add a new free storage block in the free balance tree and obtain the free storage block identifier of the new free storage block.

[0099] In one possible design, each of the multiple fragment files carries a fragment number; correspondingly, the fragment storage unit 3052 is specifically used for: storing the first fragment file among the multiple fragment files according to the fragment number, and obtaining the file identifier and the offset of the first fragment file; for each remaining fragment file, performing the following steps in sequence according to the fragment number to complete the storage: storing the current fragment file according to the file identifier and the offset of the previous fragment file, and obtaining the offset of the current fragment file.

[0100] In one possible design, the file storage processing device further includes: a verification module, used by the server to verify the current segment file, and if the verification passes, the current segment file is sent to the storage server for storage; correspondingly, after obtaining the offset of the current segment file, the device further includes: the storage server sending the file identifier and the offset of the current segment file to the server; and the server saving the hash value of the file, the file identifier, the offset of the current segment file, and the segment number of each segment file to the cache.

[0101] In one possible design, the verification module is also used by the server to obtain the maximum fragment number corresponding to the hash value in the cache. If the maximum fragment number is less than the fragment number of the current fragment file, the verification passes and the current fragment file is sent to the storage server for storage.

[0102] In one possible design, the file storage processing device further includes: a fourth sending module, used to obtain the file identifier of the file if the server determines that a hash value exists in the database; and the server sends the file identifier to the client.

[0103] The file storage processing apparatus provided in this application embodiment can be used to execute the technical solution of the file storage processing method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0104] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Each module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0105] Figure 4 This is a schematic diagram of the storage platform provided in an embodiment of this application. Figure 4 As shown, the storage platform may include: transceiver 401, processor 402, and memory 403.

[0106] Processor 402 executes computer execution instructions stored in memory, causing processor 402 to perform the scheme in the above embodiments. Processor 402 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0107] The memory 403 is connected to the processor 402 via the system bus and completes communication between them. The memory 403 is used to store computer program instructions.

[0108] The system bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus. The system bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figures, but this does not indicate that there is only one bus or a single type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write databases, and read-only databases).

[0109] The memory may include random access memory (RAM) and may also include non-volatile memory (VNM).

[0110] This application also provides a chip for executing instructions, which is used to execute the technical solution of the file storage processing method in the above embodiments.

[0111] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the file storage processing method described in the above embodiments.

[0112] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the file storage processing method in the above embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0114] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0115] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0116] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0117] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0118] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0119] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing file storage, characterized in that, The method is applied to a storage platform, which includes a server, a tracking server, and a storage server. The server receives an upload request from the client for a file to be processed, wherein the upload request carries the hash value and storage parameters of the file to be processed; The server determines whether the hash value exists in the database; If the server determines that the hash value does not exist in the database, the server sends the upload request to the tracking server. The tracking server obtains the IP address and port number of the storage server according to the upload request, and sends the IP address and port number to the server. The server sends the file to be processed to the storage server for storage based on the IP address and port number; the storage server performs the following steps on the file to be processed during the storage process: If the storage parameter is determined to be less than the merged storage threshold, then the free storage block identifier is obtained through the free balance tree, and the file to be processed is stored in the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the files to be processed whose storage parameter is less than the merged storage threshold. If it is determined that the storage parameter is greater than the fragment storage threshold, and multiple fragment files are received from the server, then the multiple fragment files are stored sequentially; wherein the multiple fragment files are obtained by the client fragmenting the file to be processed and sending the fragmented file to be processed to the server. After the storage server completes the storage of the file to be processed, it sends the file identifier of the file to be processed to the client, wherein the file identifier identifies the file storage path of the file to be processed.

2. The method according to claim 1, characterized in that, The step of obtaining the free storage block identifier through the free balanced tree includes: Determine whether there are any free storage blocks in the free balanced tree; If it is determined that the free storage block exists in the free balanced tree, then the free storage block identifier of the free balanced tree is obtained; If it is determined that the free storage block does not exist in the free balance tree, a new free storage block is added to the free balance tree, and the free storage block identifier of the new free storage block is obtained.

3. The method according to claim 1, characterized in that, Each of the multiple fragment files carries a fragment sequence number; Accordingly, the sequential storage of the plurality of fragmented files includes: Based on the fragment sequence number, the first fragment file among the plurality of fragment files is stored, and the file identifier of the file and the offset of the first fragment file are obtained; For each remaining file segment, the following steps are performed sequentially according to the segment number to complete the storage: the current file segment is stored based on the file identifier and the offset of the previous file segment, and the offset of the current file segment is obtained.

4. The method according to claim 3, characterized in that, Before storing the current file segment, the process also includes: The server verifies the current fragment file. If the verification passes, the current fragment file is sent to the storage server for storage. Accordingly, after obtaining the offset of the current file segment, the process further includes: The storage server sends the file identifier and the offset of the current fragment file to the server. The server saves the hash value of the file, the file identifier, the offset of the current fragment file, and the fragment number of each fragment file into the cache.

5. The method according to claim 4, characterized in that, The server verifies the current fragment file. If the verification passes, the server sends the current fragment file to the storage server for storage, including: The server obtains the maximum fragment number corresponding to the hash value in the cache. If the maximum fragment number is less than the fragment number of the current fragment file, the verification passes, and the current fragment file is sent to the storage server for storage.

6. The method according to any one of claims 1-5, characterized in that, After the server determines whether the hash value exists in the database, it also includes: If the server determines that the hash value exists in the database, it obtains the file identifier of the file. The server sends the file identifier to the client.

7. A file storage processing apparatus, characterized in that, The device is applied to a storage platform, which includes a server, a tracking server, and a storage server. The device includes: The upload module is used for the server to receive an upload request for a file to be processed sent by the client, wherein the upload request carries the hash value and storage parameters of the file to be processed; The judgment module is used by the server to determine whether the hash value exists in the database; The first sending module is configured to send the upload request to the tracking server if the server determines that the hash value does not exist in the database. The second sending module is used for the tracking server to obtain the IP address and port number of the storage server according to the upload request, and to send the IP address and port number to the server. A storage module is used by the server to send the file to be processed to the storage server for storage based on the IP address and the port number; the storage server performs the following steps on the file to be processed during the storage process; The storage module includes: A merge storage unit is used to obtain a free storage block identifier through a free balance tree if it is determined that the storage parameter is less than the merge storage threshold, and to store the file to be processed into the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the files to be processed whose storage parameter is less than the merge storage threshold. The fragmented storage unit is used to store the multiple fragmented files sequentially if it is determined that the storage parameter is greater than the fragmented storage threshold and multiple fragmented files are received from the server; wherein the multiple fragmented files are obtained by the client fragmenting the file to be processed and sending the fragmented file to be processed to the server. The third sending module is used to send the file identifier of the file to be processed to the client after the storage server has completed storing the file to be processed, wherein the file identifier identifies the file storage path of the file to be processed.

8. A storage platform, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images