File storage processing method and device, storage platform, medium and product

Abstract

The invention provides a file storage processing method and device, a storage platform, a medium and a product. Relates to the field of data processing. The storage platform comprises a server side, a tracking server and a storage server, and the method comprises the steps that the server side receives an uploading request for a to-be-processed file sent by a client side, and if a hash value of the to-be-processed file does not exist in a database, the uploading request is sent to the tracking server; the tracking server obtains the IP address and the port number of the storage server and sends the IP address and the port number to the server; the server sends the to-be-processed file to a storage server for storage according to the IP address and the port number; if the storage parameter is smaller than the combined storage threshold value, storing the to-be-processed file to an idle storage block; and if the storage parameter is greater than the fragment storage threshold and a plurality of fragment files sent by the server are received, sequentially storing the plurality of fragment files. According to the method, waste of storage space is avoided, the transmission efficiency is improved, and the transmission time is shortened.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device, storage platform, medium and product for processing file storage. Background Art

[0002] With the development of informatization, the total amount of data and information generated by all walks of life has exploded. When faced with files of different sizes, how to adopt appropriate storage strategies to store files has become an urgent problem to be solved.

[0003] At present, in the prior art, a traditional single-machine storage method is usually adopted to directly store files of different data sizes.

[0004] However, this method of the prior art does not distinguish the data size of the file. On the one hand, when storing files with smaller data, storage space will be wasted; on the other hand, when storing files with larger data, the transmission efficiency during the storage process will be reduced and the transmission time will be increased. Summary of the invention

[0005] The present application provides a file storage processing method, device, storage platform, medium and product to solve the problems of wasting storage space, reducing transmission efficiency during storage, and increasing transmission time in the prior art.

[0006] In a first aspect, the present application provides a method for processing file storage, comprising: a server receives an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed; the server determines whether the hash value exists in a database; if the server determines that the hash value does not exist in the database, the server sends an upload request to a tracking server; the tracking server obtains an IP address and a port number of a storage server according to the upload request, and sends the IP address and the port number to the server; the server sends the file to be processed to the storage server for storage according to the IP address and the port number; the storage server performs the following steps on the file to be processed during the storage process: if it is determined that the storage parameter is less than the specified value, the server sends the file to be processed to the storage server for storage; and storage threshold, then obtain the free storage block identifier through the free balancing tree, and store the file to be processed in the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the files to be processed whose storage parameters are less than the merge storage threshold; if it is determined that the storage parameter is greater than the shard storage threshold, and multiple shard files sent by the server are received, the multiple shard files are stored in sequence; wherein the multiple shard files are obtained by the client sharding the file to be processed and sending the sharded file to be processed to the server; after the storage server completes the storage of the file to be processed, it sends the file identifier of the file to be processed to the client, wherein the file identifier identifies the file storage path of the file to be processed.

[0007] In a possible design, obtaining a free storage block identifier through an idle balanced tree includes: determining whether there is a free storage block in the idle balanced tree; if it is determined that there is a free storage block in the idle balanced tree, obtaining the free storage block identifier of the idle balanced block; if it is determined that there is no free storage block in the idle balanced tree, adding a new free storage block in the idle balanced tree, and obtaining the free storage block identifier of the new free storage block.

[0008] In one possible design, each of the multiple shard files carries a shard sequence number; accordingly, the multiple shard files are stored in sequence, including: according to the shard sequence number, the first shard file among the multiple shard files is stored, and the file identifier of the file and the offset of the first shard file are obtained; for each of the remaining shard files, the following steps are performed in sequence according to the shard sequence number to complete the storage: according to the file identifier and the offset of the previous shard file, the current shard file is stored, and the offset of the current shard file is obtained.

[0009] In one possible design, before storing the current shard file, it also includes: the server verifies the current shard file, and if the verification passes, sends the current shard file to the storage server for storage; accordingly, after obtaining the offset of the current shard file, it also includes: the storage server sends the file identifier and the offset of the current shard file to the server; the server saves the file hash value, file identifier, the offset of the current shard file and the shard sequence number of each shard file in the cache.

[0010] In one possible design, the server verifies the current shard file, and if the verification passes, sends the current shard file to the storage server for storage, including: the server obtains the maximum shard sequence number corresponding to the hash value in the cache, if the maximum shard sequence number is smaller than the shard sequence number of the current shard file, the verification passes, and the current shard file is sent to the storage server for storage.

[0011] In a possible design, after the server determines whether the hash value exists in the database, the process further includes: if the server determines that the hash value exists in the database, obtaining a file identifier of the file; and the server sends the file identifier to the client.

[0012] In a second aspect, the present application provides a file storage processing device, comprising:

[0013] The upload module is used for receiving, at the server side, an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed;

[0014] The judgment module is used by the server to judge whether the hash value exists in the database;

[0015] A first sending module, configured to send an upload request to a tracking server if the server determines that the hash value does not exist in the database;

[0016] The second sending module is used to track the server to obtain the IP address and port number of the storage server according to the upload request, and send the IP address and port number to the server;

[0017] The storage module is used for the server to send the to-be-processed file to the storage server for storage according to the IP address and port number; the storage server performs the following steps on the to-be-processed file during the storage process;

[0018] The storage module includes:

[0019] A merging storage unit is used to obtain a free storage block identifier through an idle balancing tree if it is determined that the storage parameter is less than the merging storage threshold, and store the to-be-processed file in a free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the to-be-processed file whose storage parameter is less than the merging storage threshold;

[0020] A shard storage unit is used to store the multiple shard files in sequence if it is determined that the storage parameter is greater than the shard storage threshold and multiple shard files sent by the server are received; wherein the multiple shard files are obtained by the client sharding the to-be-processed file and sending the sharded to-be-processed file to the server;

[0021] The third sending module is used to send a file identifier of the to-be-processed file to the client after the storage server completes storing the to-be-processed file, wherein the file identifier identifies a file storage path of the to-be-processed file.

[0022] In a third aspect, an embodiment of the present application provides a storage platform, comprising: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the file storage processing method described in the first aspect and various possible designs of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the file storage processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the file storage processing method described in the first aspect and various possible designs of the first aspect.

[0025] The present application provides a method, device, storage platform, medium and product for processing file storage. If the storage parameter of a file is less than the combined storage threshold, it means that the file is a small file, and the file is stored in the free storage block after the storage unit is divided, and then merged with the files in other free storage blocks for storage, so as to avoid storing the file independently in a storage unit, resulting in a waste of storage space; if the storage parameter is greater than the fragmented storage threshold, it means that the file is a large file, and the file is fragmented into multiple fragmented files for storage, which can improve the transmission efficiency during the storage process and reduce the transmission time. In addition, by judging the hash value of the file to be processed, it is possible to avoid storing duplicate files to be processed in the storage server. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] Figure 1 An application scenario diagram of the file storage processing method provided in an embodiment of the present application;

[0028] Figure 2 A flowchart of a method for processing file storage provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of the structure of a file storage processing device provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the structure of the storage platform provided in an embodiment of the present application.

[0031] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0033] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0034] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0035] With the development of informatization, the total amount of data and information generated by various industries has exploded. When faced with files of different sizes, how to adopt appropriate storage strategies to store files has become an urgent problem to be solved. At present, in the prior art, the traditional single-machine storage method is usually adopted to directly store files of different sizes. However, this method of the prior art does not distinguish the data size of the file. On the one hand, when storing files with smaller data, it will waste storage space; on the other hand, when storing files with larger data, it will reduce the transmission efficiency during the storage process and increase the transmission time.

[0036] In response to the above technical problems, the present application proposes the following technical concept: Considering that if the data size of the file is not distinguished, on the one hand, storage space will be wasted; on the other hand, the transmission efficiency during the storage process will be reduced and the transmission time will be increased. The inventor thought of distinguishing files according to data size, merging and storing files with smaller data, and storing files with larger data in fragments. If the storage parameter of the file is less than the merged storage threshold, it means that the file is a file with smaller data, and the file is stored in the free storage block after the storage unit is divided, and then merged and stored with files in other free storage blocks to avoid storing the file independently in a storage unit, resulting in a waste of storage space; if the storage parameter is greater than the fragmented storage threshold, it means that the file is a file with larger data, and the file is fragmented into multiple fragmented files for storage, which can improve the transmission efficiency during the storage process and reduce the transmission time.

[0037] Figure 1 This is an application scenario diagram of the file storage processing method provided in the embodiment of the present application. Figure 1 As shown, the application scenario includes: a client 10 and a storage platform 20.

[0038] The storage platform 20 includes: a server 201 , a tracking server 202 and a storage server 203 .

[0039] refer to Figure 1The client 10 is used to send an upload request of the to-be-processed file to the server 201. The storage server 203 is used to periodically feed back information of the storage server 203 to the tracking server 202, including the IP address and the port number; the tracking server 202 is used to send the IP address and the port number to the server 201; the server 201 is used to send the to-be-processed file to the storage server 203 for storage according to the IP address and the port number.

[0040] Figure 2 Flow chart of the file storage processing method provided in the embodiment of the present application; the execution subject of this embodiment can be Figure 1 The storage platform in the embodiment shown includes a server, a tracking server and a storage server, and may also be other devices with similar functions, which are not particularly limited in this embodiment. Figure 2 As shown, the method includes:

[0041] S201: The server receives an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed.

[0042] The storage parameter indicates the data size of the file to be processed.

[0043] In this embodiment, the hash value of the file to be processed is calculated by the client through a hash algorithm.

[0044] Optionally, the hash algorithm may be Message-Digest Algorithm 5 (MD5), where MD5 is a type of hash algorithm, and the hash value may be an MD5 value.

[0045] S202: The server determines whether there is a hash value in the database.

[0046] S203: If the server determines that the hash value does not exist in the database, the server sends an upload request to the tracking server.

[0047] In this embodiment, if the hash value does not exist in the database, it means that the to-be-processed file corresponding to the hash value has not been uploaded.

[0048] Optionally, if the server determines that the hash value exists in the database, a file identifier of the file is obtained; the server sends the file identifier to the client.

[0049] In this embodiment, if there is a hash value in the database, it means that the file to be processed has been stored in the storage server, and the server sends a file identifier request to the tracking server, the tracking server sends the file identifier to the server, the server sends the file identifier to the client, and the client notifies the user that the upload is successful.

[0050] S204: The tracking server obtains the IP address and port number of the storage server according to the upload request, and sends the IP address and port number to the server.

[0051] Optionally, the storage server periodically sends the operation status of the storage server and some important information to the tracking server, such as the number of uploaded files, the number of deleted files, the total number of downloads and the remaining memory space and file synchronization information in the storage server.

[0052] In this embodiment, the tracking server obtains the available IP address and port number in the storage server according to the upload request.

[0053] S205: The server sends the to-be-processed file to the storage server for storage according to the IP address and port number; the storage server performs the following steps on the to-be-processed file during the storage process:

[0054] In this embodiment, the server communicates with the storage server according to the IP address and port number, and sends the to-be-processed file to the storage server for storage.

[0055] Specifically, step S205 includes S2051-S2052:

[0056] S2051: If it is determined that the storage parameter is less than the merge storage threshold, the free storage block identifier is obtained through the free balance tree, and the file to be processed is stored in the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the file to be processed whose storage parameter is less than the merge storage threshold.

[0057] The storage space in the storage server is divided into a plurality of Trunk blocks, each of which may be 64 MB in size, and each Trunk block is divided into a plurality of free storage blocks, each of which may be 16 MB in size.

[0058] Optionally, the merge storage threshold is 16 MB. When the storage parameter is less than 16 MB, the to-be-processed file will be stored in a free storage block and merged with other to-be-processed files for storage.

[0059] In this embodiment, after storing the to-be-processed file in the corresponding free storage block, the storage server stores the file identifier, Trunk ID, the offset of the to-be-processed file in the Trunk, the merge storage time, etc. The file identifier identifies the file storage path of the to-be-processed file in the storage server.

[0060] Specifically, the process of obtaining the free storage block identifier through the free balance tree will be introduced in detail in the subsequent embodiments.

[0061] S2052: If it is determined that the storage parameter is greater than the shard storage threshold, and multiple shard files are received from the server, the multiple shard files are stored in sequence; wherein the multiple shard files are obtained by the client sharding the file to be processed and sending the sharded file to be processed to the server.

[0062] Optionally, the shard storage threshold may be 100MB.

[0063] In this embodiment, if the storage parameter of the file to be processed is greater than the fragment storage threshold, the file to be processed needs to be stored in fragments.

[0064] In this embodiment, when the client sends a request to upload a file to be processed, if the storage parameter is greater than the shard storage threshold, the file to be processed is sharded according to the shard size to obtain multiple shard files, each of which carries a shard sequence number. The client sends the multiple shard files to the server, and the server sends them to the storage server in sequence according to the shard sequence numbers, and the storage server stores the multiple shard files in sequence.

[0065] Specifically, according to the shard sequence number, the first shard file among multiple shard files is stored, and the file identifier and the offset of the first shard file are obtained; for each remaining shard file, the following steps are performed in sequence according to the shard sequence number to complete the storage: according to the file identifier and the offset of the previous shard file, the current shard file is stored, and the offset of the current shard file is obtained.

[0066] In this embodiment, according to the file identifier of the file and the offset of the first slice file, the second slice file is stored according to the slice sequence number, and the offset of the second slice file is obtained; according to the file identifier of the file and the offset of the second slice file, the third slice file is stored according to the slice sequence number, and the offset of the third slice file is obtained, and so on, until the offset of the second-to-last slice file is obtained, and the last slice file is stored according to the file identifier of the file and the offset of the second-to-last slice file to complete the storage of multiple slice files.

[0067] The offset is used to confirm the location where each shard file is stored.

[0068] Optionally, if it is determined that the storage parameter is greater than the merge storage threshold and less than the shard storage threshold, the file to be processed is directly saved.

[0069] S206: After the storage server finishes storing the file to be processed, it sends a file identifier of the file to be processed to the client, where the file identifier identifies a file storage path of the file to be processed.

[0070] In this embodiment, after the storage server completes storing the pending file, it sends the file identifier of the pending file to the server, and the server persists the file identifier and sends it to the client to notify the user that the upload is successful.

[0071] In summary, if the storage parameter of a file is less than the merge storage threshold, it means that the file is a small file, and the file is stored in the free storage block after the storage unit is divided, and then merged with the files in other free storage blocks to avoid storing the file independently in a storage unit, resulting in a waste of storage space; if the storage parameter is greater than the fragment storage threshold, it means that the file is a large file, and the file is fragmented into multiple fragment files for storage, which can improve the transmission efficiency during the storage process and reduce the transmission time. In addition, by judging the hash value of the file to be processed, it is possible to avoid storing duplicate files to be processed in the storage server.

[0072] Based on the above embodiment, in this embodiment, the process of obtaining the free storage block identifier through the free balance tree is introduced in detail, as follows:

[0073] S301: Determine whether there is an idle storage block in the idle balance tree.

[0074] In this embodiment, the storage server divides the storage space into multiple Trunk blocks, and each Trunk is divided into multiple free storage blocks. Trunk uses the method of pre-allocating storage space to avoid repeated allocation of storage space. In order to better manage the free storage space, the storage server constructs an idle balance tree with free storage space as the keyword, and stores free storage blocks of the same size in the form of a linked list. Optionally, the storage server does not have a storage server cluster, and not all storage servers have the ability to manage and allocate free storage space. If the storage server that receives the file to be processed has a node that manages and allocates free storage space, it is determined whether there is a free storage block in the idle balance tree.

[0075] Optionally, the basis for judging whether there is a free storage block in the free balance tree is that the free storage block is larger than and closest to the storage parameter of the file to be processed.

[0076] S302: If it is determined that there is an idle storage block in the idle balancing tree, the idle storage block identifier of the idle balancing block is obtained.

[0077] S303: If it is determined that there is no free storage block in the free balance tree, a new free storage block is added to the free balance tree, and a free storage block identifier of the new free storage block is obtained.

[0078] In this embodiment, if there is no free storage block, a new Trunk block is created, and the new Trunk block is divided into free storage blocks and added to the free balance tree.

[0079] In summary, the idle balanced tree can better manage the idle storage space of the storage server and further avoid wasting storage space.

[0080] Based on the above embodiment, in this embodiment, the processing before and after each shard file is stored is described in detail, as follows:

[0081] The server verifies the current shard file, and if the verification passes, sends the current shard file to the storage server for storage.

[0082] Specifically, the server obtains the maximum fragment sequence number corresponding to the hash value in the cache. If the maximum fragment sequence number is smaller than the fragment sequence number of the current fragment file, the verification passes and the current fragment file is sent to the storage server for storage.

[0083] In this embodiment, the information of the current shard file is saved in the cache in the form of a key-value pair. The key is a hash value, and the value includes a file identifier, an offset, and a maximum shard sequence number. The server obtains the maximum shard sequence number based on the hash value of the file to be processed. If the maximum shard sequence number is less than the shard sequence number of the current shard file, it means that the current shard file has not been stored, and the verification passes.

[0084] Optionally, if the maximum fragment sequence number is greater than the fragment sequence number of the current fragment file, it means that the current fragment file has been stored, and the verification fails, and the current fragment file is skipped. Even in the case of network fluctuations or interruptions, the accuracy of fragment file upload can be ensured.

[0085] Optionally, after obtaining the offset of the current shard file, the storage server sends the file identifier and the offset of the current shard file to the server; the server saves the file hash value, file identifier, the offset of the current shard file and the shard sequence number of each shard file in the cache.

[0086] The offset is used to confirm the location where each shard file is stored.

[0087] In summary, by verifying the current segment file before storing it, on the one hand, it is possible to avoid uploading duplicate segment files; on the other hand, if there is a network fluctuation or interruption, the segment file can be resumed based on the cache.

[0088] Figure 3 A schematic diagram of the structure of a file storage processing device provided in an embodiment of the present application. Figure 3As shown, the file storage processing device includes: an upload module 301, a judgment module 302, a first sending module 303, a second sending module 304, a storage module 305 and a third sending module 306. The storage module 305 includes: a merge storage unit 3051 and a fragment storage unit 3052.

[0089] Upload module 301, used for receiving, at the server end, an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed;

[0090] A judgment module 302 is used for the server to judge whether there is a hash value in the database;

[0091] A first sending module 303, configured to send an upload request to a tracking server if the server determines that the hash value does not exist in the database;

[0092] The second sending module 304 is used for the tracking server to obtain the IP address and port number of the storage server according to the upload request, and send the IP address and port number to the server;

[0093] Storage module 305, used for the server to send the to-be-processed file to the storage server for storage according to the IP address and port number; the storage server performs the following steps on the to-be-processed file during the storage process;

[0094] Wherein, the storage module 305 includes:

[0095] The merge storage unit 3051 is used to obtain the free storage block identifier through the free balance tree if it is determined that the storage parameter is less than the merge storage threshold, and store the to-be-processed file in the free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the to-be-processed file whose storage parameter is less than the merge storage threshold;

[0096] The shard storage unit 3052 is used to store the multiple shard files in sequence if it is determined that the storage parameter is greater than the shard storage threshold and multiple shard files sent by the server are received; wherein the multiple shard files are obtained by the client sharding the to-be-processed file and sending the sharded to-be-processed file to the server;

[0097] The third sending module 306 is configured to send a file identifier of the file to be processed to the client after the storage server completes storing the file to be processed, wherein the file identifier identifies a file storage path of the file to be processed.

[0098] In a possible design, the merged storage unit 3051 is specifically used to: determine whether there is a free storage block in the free balanced tree; if it is determined that there is a free storage block in the free balanced tree, obtain the free storage block identifier of the free balanced block; if it is determined that there is no free storage block in the free balanced tree, add a new free storage block to the free balanced tree, and obtain the free storage block identifier of the new free storage block.

[0099] In one possible design, each of the multiple shard files carries a shard sequence number; accordingly, the shard storage unit 3052 is specifically used to: store the first shard file among the multiple shard files according to the shard sequence number, and obtain the file identifier of the file and the offset of the first shard file; for each remaining shard file, perform the following steps in sequence according to the shard sequence number to complete the storage: store the current shard file according to the file identifier and the offset of the previous shard file, and obtain the offset of the current shard file.

[0100] In one possible design, the file storage processing device also includes: a verification module, which is used by the server to verify the current shard file, and if the verification passes, the current shard file is sent to the storage server for storage; accordingly, after obtaining the offset of the current shard file, it also includes: the storage server sends the file identifier and the offset of the current shard file to the server; the server saves the file hash value, file identifier, the offset of the current shard file and the shard sequence number of each shard file in the cache.

[0101] In one possible design, the verification module is also used by the server to obtain the maximum shard number corresponding to the hash value in the cache. If the maximum shard number is smaller than the shard number of the current shard file, the verification passes and the current shard file is sent to the storage server for storage.

[0102] In one possible design, the file storage processing device also includes: a fourth sending module, used to obtain a file identifier of the file if the server determines that the hash value exists in the database; the server sends the file identifier to the client.

[0103] The file storage processing device provided in the embodiment of the present application can be used to execute the technical solution of the file storage processing method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0104] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. Each module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0105] Figure 4 This is a schematic diagram of the structure of the storage platform provided in the embodiment of the present application. Figure 4 As shown, the storage platform may include: a transceiver 401 , a processor 402 , and a memory 403 .

[0106] The processor 402 executes the computer-executable instructions stored in the memory, so that the processor 402 executes the solution in the above embodiment. The processor 402 can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0107] The memory 403 is connected to the processor 402 via a system bus and completes communication between them. The memory 403 is used to store computer program instructions.

[0108] The system bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries).

[0109] The memory may include random access memory (RAM) and may also include non-volatile memory (VNM).

[0110] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the file storage processing method in the above embodiment.

[0111] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the file storage processing method of the above embodiment.

[0112] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the file storage processing method in the above embodiment.

[0113] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0114] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0115] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0116] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0117] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0118] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic control unit or a main control device.

[0119] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing file storage, characterized in that: Applied to a storage platform, the storage platform includes a service end, a tracking server and a storage server, the method includes: The server receives an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed; The server determines whether the hash value exists in the database; If the server determines that the hash value does not exist in the database, the server sends the upload request to the tracking server; The tracking server obtains the IP address and port number of the storage server according to the upload request, and sends the IP address and port number to the service end; The server sends the to-be-processed file to the storage server for storage according to the IP address and the port number; the storage server performs the following steps on the to-be-processed file during the storage process: If it is determined that the storage parameter is less than the merge storage threshold, a free storage block identifier is obtained through an idle balancing tree, and the to-be-processed file is stored in a free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the to-be-processed file whose storage parameter is less than the merge storage threshold; If it is determined that the storage parameter is greater than the shard storage threshold, and multiple shard files sent by the server are received, the multiple shard files are stored in sequence; wherein the multiple shard files are obtained by the client sharding the file to be processed and sending the sharded files to be processed to the server; After the storage server completes storing the file to be processed, it sends a file identifier of the file to be processed to the client, wherein the file identifier identifies a file storage path of the file to be processed.

2. The method according to claim 1, characterized in that: The step of obtaining the free storage block identifier through the free balance tree includes: Determine whether there is an idle storage block in the idle balanced tree; If it is determined that the free storage block exists in the free balancing tree, obtaining a free storage block identifier of the free balancing block; If it is determined that the free storage block does not exist in the free balancing tree, a new free storage block is added to the free balancing tree, and a free storage block identifier of the new free storage block is obtained.

3. The method according to claim 1, characterized in that Each of the multiple fragment files carries a fragment sequence number; Accordingly, the storing of the plurality of fragment files in sequence includes: According to the fragment sequence number, the first fragment file among the multiple fragment files is stored, and the file identifier of the file and the offset of the first fragment file are obtained; For each remaining shard file, the following steps are performed in sequence according to the shard sequence number to complete the storage: according to the file identifier and the offset of the previous shard file, the current shard file is stored, and the offset of the current shard file is obtained.

4. The method according to claim 3, characterized in that Before storing the current fragment file, the method further includes: The server verifies the current segment file, and if the verification passes, sends the current segment file to the storage server for storage; Accordingly, after obtaining the offset of the current shard file, the method further includes: The storage server sends the file identifier and the offset of the current shard file to the server; The server saves the hash value of the file, the file identifier, the offset of the current fragment file and the fragment sequence number of each fragment file in a cache.

5. The method according to claim 4, characterized in that The server verifies the current segment file, and if the verification passes, sends the current segment file to the storage server for storage, including: The server obtains the maximum fragment sequence number corresponding to the hash value in the cache. If the maximum fragment sequence number is smaller than the fragment sequence number of the current fragment file, the verification passes and the current fragment file is sent to the storage server for storage.

6. The method according to any one of claims 1 to 5, characterized in that: After the server determines whether the hash value exists in the database, the method further includes: If the server determines that the hash value exists in the database, obtaining a file identifier of the file; The server sends the file identifier to the client.

7. A file storage processing device, characterized in that: Applied to a storage platform, the storage platform includes a service end, a tracking server and a storage server, and the device includes: An upload module, configured to receive, at the server end, an upload request for a file to be processed sent by a client, wherein the upload request carries a hash value and storage parameters of the file to be processed; A judgment module, used by the server to judge whether the hash value exists in the database; A first sending module, configured to send the upload request to the tracking server if the server determines that the hash value does not exist in the database; A second sending module, configured for the tracking server to obtain the IP address and port number of the storage server according to the upload request, and send the IP address and port number to the server; A storage module is used for the server to send the to-be-processed file to the storage server for storage according to the IP address and the port number; the storage server performs the following steps on the to-be-processed file during the storage process; Wherein, the storage module includes: A merging storage unit, configured to obtain a free storage block identifier through an idle balancing tree if it is determined that the storage parameter is less than a merging storage threshold, and store the to-be-processed file in a free storage block corresponding to the free storage block identifier; wherein the free storage block is used to merge and store the to-be-processed file whose storage parameter is less than the merging storage threshold; A shard storage unit is configured to store the multiple shard files in sequence if it is determined that the storage parameter is greater than a shard storage threshold and multiple shard files sent by the server are received; wherein the multiple shard files are obtained by the client sharding the to-be-processed file and sending the sharded to-be-processed files to the server; The third sending module is used to send the file identifier of the file to be processed to the client after the storage server completes storing the file to be processed, wherein the file identifier identifies the file storage path of the file to be processed.

8. A storage platform, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images