A mass file storage method, device, equipment and storage medium

Abstract

The application provides a mass file storage method, device and equipment and a storage medium, and relates to the technical field of data processing. The application constructs a linked list cache of file storage into the memory of each storage server; receives a storage request of one or more clients for a to-be-stored file to a server end; calculates the fragmentation strength of each storage server in sequence in response to the storage request; screens out a storage server with balanced fragmentation strength as a balanced server; and stores the to-be-stored file into the balanced server with the minimum fragmentation strength. The application guarantees the local stability storage requirement of file data and improves the storage efficiency. The application can dynamically judge the balancing of the storage server during use, thereby avoiding the problem of excessive file fragmentation of local servers. When high-frequency and multiple operations are performed on the storage server, the probability of problems such as high delay, downtime, crash or long-time loss of response of the server is reduced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a method, apparatus, device, and storage medium for storing massive amounts of files. Background Technology

[0002] Currently, file storage and transmission are mainly based on disk file systems such as local disk files, distributed file systems, and object storage systems. These systems are primarily designed for storing large files rather than small, fragmented files. However, in various business scenarios, such as OA office systems, process management systems, and workflow systems, a large number of small files are often transmitted and exchanged in various processes. When these systems are used for transmission and storage, the storage and transmission speed is often significantly reduced.

[0003] Therefore, there is an urgent need to improve the existing methods for storing massive files, so as to improve transmission speed, response speed, data locality, and storage efficiency of massive files, and avoid the problem of server overload and unresponsiveness caused by high-frequency operations. Summary of the Invention

[0004] The purpose of this invention is to provide a method, apparatus, device and storage medium for storing massive amounts of files, so as to solve one or more technical problems existing in the prior art, or at least provide a beneficial option or create conditions.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for storing massive amounts of files is provided, the method comprising the following steps:

[0006] S100, construct all file storage structure information in each storage server into a linked list and cache it in the memory of the corresponding storage server; wherein, the file storage structure information includes file header structure information and file data storage area information;

[0007] S200, receiving storage requests for files to be stored from one or more clients to the server; wherein, the storage request includes file storage structure information of the file to be stored;

[0008] S300, in response to storage requests, calculates the fragmentation intensity of each storage service server in turn;

[0009] S400 selects storage servers whose fragmentation intensity has reached a balanced level as the load balancer.

[0010] S500 stores the files to be stored on the load balancer with the lowest fragmentation intensity value.

[0011] In some embodiments, the method further includes the step of: S600, selecting storage servers whose fragmentation intensity has not reached the balance as unbalanced servers, and performing fragmentation synchronization operation or disk defragmentation on each unbalanced server; wherein, the fragmentation synchronization operation is to form a cluster of all unbalanced servers and perform load balancing on the files stored in the cluster using the FastDFS load balancing method.

[0012] In some embodiments, in S100, constructing all file storage structure information from each storage server into a linked list and caching it in the memory of the corresponding storage server includes:

[0013] In each storage server, the file storage structure information is used as nodes. The nodes are then linked together in descending order of any one of the following: the number of accesses, queries, modifications, or downloads of the files stored on each node within the storage server. This results in a linked list of data storage structures, which is then cached in the memory of the respective storage servers. This allows the most frequently accessed files to be placed at the head of the list, and the least frequently accessed files at the tail.

[0014] In other embodiments, in S100, the step of constructing a linked list of all file storage structure information in each storage server and caching it in the memory of the corresponding storage server includes:

[0015] In each storage server, the storage structure information of each file is used as a node. The storage structure obtained by connecting the nodes in the order of their storage time in the storage server is used as a linked list. The linked list is then cached in the memory of the corresponding storage server.

[0016] In some embodiments, in S200, the files to be stored are text files, image files, audio files, or video files uploaded by various clients. The server is interconnected with multiple storage servers, which are IDC servers or tracking servers of the FastDFS file system.

[0017] Preferably, if the client accesses, queries, modifies, or downloads the files corresponding to the nodes of the linked lists in the memory of each storage server, the number of accesses, queries, modifications, or downloads of the corresponding stored files is recorded.

[0018] Furthermore, in S300, the method for sequentially calculating the fragmentation intensity of each storage service server is as follows:

[0019] Let SeCN be the number of storage servers, and Ut be the access frequency of each node in the linked list corresponding to each storage server. The access frequency Ut is any one of the following: the number of accesses, queries, modifications, or downloads of the file stored in the storage server corresponding to the file storage structure information of the node; let i1 represent the sequence number of the storage server, i1∈[1,SeCN].

[0020] In some embodiments, the centralized access trend value CE(i1) of the i1th storage server is calculated by traversing i1 within the range [1, SeCN] using the following formula:

[0021]

[0022] Where i2 is a variable, N(i1) is the number of nodes in the linked list corresponding to the i1th storage server, MaxUt(i1,i2) is the maximum value of the access frequency of the nodes from the 1st to the i2th in the linked list corresponding to the i1th storage server, and MeanUt(i1,i2) is the average value of the access frequency of the nodes from the 1st to the i2th in the linked list corresponding to the i1th storage server.

[0023] The above formula uses the average of the differences between the local extrema and the mean of the access frequency between each storage server and its neighboring storage servers as the concentrated access trend value to represent the access popularity relationship of files in each storage server. However, when multiple storage servers are about to generate small file fragments (small file fragments are a large number of discontinuous discrete files generated in the storage medium), although the final value is the average, it will still affect the accuracy of the access popularity identification and the subsequent judgment of the fragmentation intensity of the storage server. Therefore, the concentrated access trend value CE(i1) calculated by the following formula is multiplied by a coefficient in the logarithmic form of the difference trend of all access frequencies, thereby improving the accuracy of the concentrated access trend value calculation for files in each storage server that exhibits accelerated generation of small file fragments.

[0024] In other embodiments, the centralized access trend value CE(i1) of the i1th storage server is calculated by traversing i1 within the range [1, SeCN] using the following formula:

[0025]

[0026] Where i2 is a variable, N(i1) is the number of nodes in the linked list corresponding to the i1th storage server, Ut(i2)i1 is the access frequency of the i2th node in the linked list corresponding to the i1th storage server, MaxUt(i1,i2) is the maximum value of the access frequency from the 1st to the i2th node in the linked list corresponding to the i1th storage server, MinUt(i1,i2) is the minimum value of the access frequency from the 1st to the i2th node in the linked list corresponding to the i1th storage server, and log is a logarithmic function;

[0027] In some embodiments, the fragmentation intensity of the i1th storage server is calculated by directly using the centralized access trend value CE(i1) of the i1th storage server as the fragmentation intensity of the i1th storage server.

[0028] Since the centralized access trend value CE(i1) of the i1th storage server represents the access popularity, the higher the centralized access trend value of files in the storage server, the greater the probability of generating discontinuous discrete files in the storage medium of the storage server, and the greater the fragmentation intensity in the storage server. Therefore, the centralized access trend value CE(i1) can be directly used as the fragmentation intensity. The fragmentation intensity of each storage server is then calculated in turn.

[0029] However, directly using the centralized access trend value CE(i1) as the fragmentation strength will result in significant distortion when multiple storage servers accelerate the generation of small file fragments. This is because the linked lists in each storage server rapidly add and remove nodes during the subsequent fragmentation synchronization and balancing operations. Therefore, the fragmentation strength can also be calculated using the following scheme:

[0030] Furthermore, in S400, the method for selecting storage servers whose fragmentation intensity has reached a balanced state as balanced servers is as follows: the average fragmentation intensity of all storage servers is StrMean; storage servers with fragmentation intensity greater than or equal to StrMean are marked as unbalanced servers, and storage servers with fragmentation intensity less than StrMean are marked as balanced servers; storage servers with fragmentation intensity less than StrMean are selected as balanced servers.

[0031] Furthermore, in S500, the method for storing the file to be stored in the load balancer with the smallest fragmentation intensity value is as follows: the server receives the file to be stored from the client, stores the file to be stored in the load balancer with the smallest fragmentation intensity value, and adds the file storage structure information corresponding to the file to be stored as a node of the linked list to the linked list in the memory of the load balancer.

[0032] To achieve the above objectives, a second aspect of the present invention provides a massive file storage device, the device comprising:

[0033] The first module is used to construct a linked list of all file storage structure information in each storage server and cache it in the memory of the corresponding storage server.

[0034] The second module is used to receive storage requests for files to be stored from one or more clients and send them to the server.

[0035] The third module is used to calculate the fragmentation intensity of each storage server in turn in response to storage requests;

[0036] The fourth module is used to select storage servers whose fragmentation intensity has reached a balanced level as load balancers.

[0037] The fifth module is used to store the files to be stored in the load balancer with the lowest fragmentation intensity.

[0038] To achieve the above objectives, a third aspect of the present invention provides an electronic device, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing communication between the processor and the memory, wherein the program, when executed by the processor, implements the method described in the first aspect above.

[0039] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs that can be executed by one or more processors to implement the method described in the first aspect.

[0040] According to the solution provided in the embodiments of the present invention, files in the storage server are located and stored by calculating the fragmentation intensity, which avoids the fragmentation and voids inside large files caused by long-term use in the storage server. This ensures the data locality and stable storage requirements of the files and improves storage efficiency. In particular, it can dynamically judge the storage server leveling during use to avoid the problem of excessive file fragmentation on local servers. This prevents the storage server from having a large number of discontinuous file storage fragments. When performing high-frequency and frequent access, query, modification or download operations on files in the data storage area of ​​the storage server, it reduces the probability of high latency, downtime, crash or long-term unresponsiveness of the server.

[0041] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the massive file storage method provided in the embodiments of the present invention;

[0044] Figure 2 This is a schematic diagram of the structure of the massive file storage device provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0049] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0050] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0051] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0052] Related technologies, such as the Chinese invention patent publication number CN104731886B, which describes a method and system for processing massive amounts of files, although they merge small files into large files for storage, and although they improve file retrieval efficiency by reducing the amount of metadata, and although storing small files separately creates storage fragmentation, thus reducing storage fragmentation after merging, the merging storage mechanism of this scheme, while implementing the merging storage, will result in the following two situations when storing, reading, and deleting small files: appending the file to be stored to a large file, and deleting small files from a large file and then re-merging the two separated parts; therefore, these operations will bring... The significant system overhead, especially from small file deletion operations, can lead to small files being located in the middle of the merged large file. Therefore, it is inevitable that these small files will need to be deleted and re-merged, putting enormous pressure on the storage server. When a large number of clients perform file addition and deletion operations, and if the large files to which these small files are attached have not been deleted for a long time, a large number of discontinuous storage fragments will appear in the corresponding large file data storage area. These storage fragments cannot be recycled by the system. When frequent deletion or modification operations are performed on small files in a large file data storage area, it can even cause problems such as high server latency, downtime, crashes, or prolonged unresponsiveness.

[0053] To address the problems in the background art, embodiments of the present invention provide a method, apparatus, device, and storage medium for storing massive files, which are specifically described through the following embodiments. First, the method for storing massive files in the embodiments of the present invention is described.

[0054] like Figure 1 The diagram shown is a flowchart of a method for storing massive amounts of files. The following section will combine... Figure 1 This invention describes a method for storing massive amounts of files according to an embodiment of the present invention, the method comprising the following steps:

[0055] S100, all file storage structure information in each storage server is constructed into a linked list and cached in the memory of the corresponding storage server; wherein, the file storage structure information includes file header structure information and file data storage area information;

[0056] S200, receiving storage requests for files to be stored from one or more clients to the server; wherein, the storage request includes file storage structure information of the file to be stored;

[0057] S300, in response to storage requests, calculates the fragmentation intensity of each storage service server in turn;

[0058] S400 selects storage servers whose fragmentation intensity has reached a balanced level as the load balancer.

[0059] S500 stores the files to be stored on the load balancer with the lowest fragmentation intensity value.

[0060] In the embodiments provided by this invention, files in the storage server are located and stored by calculating the fragmentation intensity, avoiding fragmentation and voids within large files caused by long-term use in the storage server. This ensures the data locality and stability of file storage requirements and improves storage efficiency. In particular, it can dynamically judge the storage server leveling during use to avoid the problem of excessive file fragmentation on local servers, so that the storage server will not have a large number of discontinuous file storage fragments. When performing high-frequency and frequent access, query, modification or download operations on files in the data storage area of ​​the storage server, the probability of high latency, downtime, crash or long-term unresponsiveness of the server is reduced.

[0061] It should be noted that when a storage request for a file to be stored is received from one or more clients, S200 to S500 are executed in a loop to achieve the beneficial effects of the present invention.

[0062] In some improved embodiments, the method further includes the step of: S600, selecting storage servers whose fragmentation intensity has not reached a balanced state as unbalanced servers, and performing fragmentation synchronization operations or disk defragmentation on each unbalanced server.

[0063] In some embodiments, the fragment synchronization operation involves forming a cluster of all unbalanced servers and using FastDFS's load balancing method to load balance the files stored in the cluster.

[0064] In some improved embodiments, in S100, the method for constructing a linked list of all file storage structure information in each storage server and caching it in the memory of the corresponding storage server is as follows: In each storage server, each file storage structure information is used as a node. The nodes are sequentially connected in descending order of any one of the following: the number of accesses, queries, modifications, or downloads of the files stored at each node in the storage server. The resulting data storage structure is then used as a linked list, and the linked list is cached in the memory of the corresponding storage server. Thus, the file with the highest client access frequency is placed at the head of the linked list, and the file with the lowest access frequency is placed at the tail of the linked list.

[0065] In some improved embodiments, in S100, the method for constructing all file storage structure information in each storage server into a linked list and caching it in the memory of the corresponding storage server is as follows: in each storage server, each file storage structure information is used as a node, and the storage structure obtained by sequentially connecting each node in the order of the node's storage time in the storage server is used as a linked list, and then the linked list is cached in the memory of the corresponding storage server.

[0066] In some embodiments, in S200, the files to be stored are text files, image files, audio files, or video files uploaded by various clients. The server is interconnected with multiple storage servers, which are IDC servers or tracking servers of the FastDFS file system.

[0067] In some preferred embodiments, if a client accesses, queries, modifies, or downloads the files corresponding to the nodes of the linked lists in the memory of each storage service server, the number of times the corresponding files are accessed, queried, modified, or downloaded is recorded.

[0068] In real-world storage server applications, files on various storage servers are constantly being added and removed (often added where files have been deleted, leaving numerous unusable storage gaps; for example, deleting a 10MB file leaves a 10MB gap, and then uploading a 9.5MB file creates an additional 0.5MB gap. If no new file smaller than 0.5MB fills this space, this 0.5MB fragment will persist). Furthermore, file locations are random. Therefore, over time, the deletion or merging of numerous files creates a large number of unusable fragmented locations on storage servers. This is especially true for frequently accessed files, which can exacerbate this phenomenon in some storage servers in the short term. Therefore, it is necessary to measure the rate at which storage servers begin to fragment. Based on the fragmentation intensity calculated from the access trends of files on each storage server, the storage servers that are rapidly generating small file fragments can be quickly identified. This allows for subsequent fragmentation balancing operations on these servers. This method provides the following steps:

[0069] In some preferred embodiments, in S300, the method for sequentially calculating the fragmentation intensity of each storage service server is as follows:

[0070] Let SeCN be the number of storage servers, and Ut be the access frequency of each node in the linked list corresponding to each storage server. The access frequency Ut is any one of the following: the number of accesses, queries, modifications, or downloads of the file stored in the storage server corresponding to the file storage structure information of the node; let i1 represent the sequence number of the storage server, i1∈[1,SeCN].

[0071] In some embodiments, the centralized access trend value CE(i1) of the i1th storage server is calculated by traversing i1 within the range [1, SeCN] using the following formula:

[0072]

[0073] Where i2 is a variable, N(i1) is the number of nodes in the linked list corresponding to the i1th storage server, MaxUt(i1,i2) is the maximum value of the access frequency of the nodes from the 1st to the i2th in the linked list corresponding to the i1th storage server, and MeanUt(i1,i2) is the average value of the access frequency of the nodes from the 1st to the i2th in the linked list corresponding to the i1th storage server.

[0074] The above formula uses the average of the differences between the local extrema and the mean of the access frequency between each storage server and its neighboring storage servers as a concentrated access trend value to represent the access popularity relationship of files on each storage server. However, when multiple storage servers are about to generate small file fragments (small file fragments are the generation of a large number of discontinuous discrete files in the storage medium), although the final value is the mean, it will still affect the accuracy of access popularity identification and affect the subsequent fragmentation intensity judgment of storage servers.

[0075] In other embodiments, the centralized access trend value CE(i1) of the i1th storage server is calculated by traversing i1 within the range [1, SeCN] using the following formula:

[0076]

[0077] Where i2 is a variable, N(i1) is the number of nodes in the linked list corresponding to the i1th storage server, Ut(i2)i1 is the access frequency of the i2th node in the linked list corresponding to the i1th storage server, MaxUt(i1,i2) is the maximum value of the access frequency from the 1st to the i2th node in the linked list corresponding to the i1th storage server, MinUt(i1,i2) is the minimum value of the access frequency from the 1st to the i2th node in the linked list corresponding to the i1th storage server, and log is a logarithmic function.

[0078] In this embodiment, the concentrated access trend value CE(i1) is obtained by multiplying it by the coefficient of the logarithmic form of the difference trend of all access frequencies, thereby improving the calculation accuracy of the concentrated access trend value of files in various storage servers that exhibit accelerated generation of small file fragments.

[0079] In some embodiments, the fragmentation intensity of the i1th storage server is calculated by directly using the centralized access trend value CE(i1) of the i1th storage server as the fragmentation intensity of the i1th storage server.

[0080] Since the centralized access trend value CE(i1) of the i1th storage server represents the access popularity, the higher the centralized access trend value of files in the storage server, the greater the probability of generating discontinuous discrete files in the storage medium of the storage server, and the greater the fragmentation intensity in the storage server. Therefore, the centralized access trend value CE(i1) can be directly used as the fragmentation intensity. The fragmentation intensity of each storage server is calculated sequentially according to the above formula.

[0081] However, directly using the centralized access trend value CE(i1) as the fragmentation intensity also affects the accuracy of the centralized access trend value of a storage server when multiple storage servers accelerate the generation of small file fragments. This is because when the storage server calculating the centralized access trend value is storing or deleting files at the same time as other storage servers, high latency, downtime, crashes, or prolonged unresponsiveness of neighboring storage servers can cause a large error in the frequency of access to each storage server by the server side.

[0082] In some preferred embodiments, the fragmentation intensity of the i1th storage server can also be calculated as follows:

[0083] Let MaxCE be the maximum value of the centralized access trend value of all storage servers, MeanCE be the average value of the centralized access trend value of all storage servers, and MinCE be the minimum value of the centralized access trend value of all storage servers. Calculate the strong trend ratio as CEup, where CEup = MeanCE / MaxCE; calculate the weak trend ratio as CEdo, where CEdo = MinCE / MeanCE.

[0084] Calculate the strength coefficient Stin(i1) of the i1th storage server:

[0085] When CEup≥CEdo

[0086] When CEup < CEdo

[0087] Where i3 is a variable, and CE(i3) is the centralized access trend value of the i3th storage server;

[0088] The product of the centralized access trend value CE(i1) and Stin(i1) of the i1th storage server is used as the fragmentation intensity of the i1th storage server.

[0089] In this embodiment, the speed trend of storage server accelerating the generation of small file fragments is measured by the strength coefficient, thereby eliminating the impact of the weakening or strengthening speed of fragment generation by other nearby storage servers on the accuracy of the fragmentation strength of the current storage server.

[0090] By multiplying by the strength coefficient Stin, we can further avoid the impact of storage servers adjacent to the i1th storage server that accelerate the generation of small file fragments on the accuracy of fragmentation strength calculation, reduce the sensitivity of fragmentation strength, and improve the stability of fragmentation strength judgment.

[0091] In some preferred embodiments, in S400, the method for selecting storage servers whose fragmentation intensity has reached a balanced state as balanced servers is as follows: the average fragmentation intensity of all storage servers is denoted as StrMean; storage servers with fragmentation intensity greater than or equal to StrMean are marked as unbalanced servers, and storage servers with fragmentation intensity less than StrMean are marked as balanced servers; storage servers with fragmentation intensity less than StrMean are selected as balanced servers.

[0092] In the embodiments provided by this invention, files in the storage server are located and stored by calculating the fragmentation intensity, avoiding fragmentation and voids within large files caused by long-term use in the storage server. This ensures the data locality and stability of file storage requirements and improves storage efficiency. In particular, it can dynamically judge the storage server leveling during use to avoid the problem of excessive file fragmentation on local servers, so that the storage server will not have a large number of discontinuous file storage fragments. When performing high-frequency and frequent access, query, modification or download operations on files in the data storage area of ​​the storage server, the probability of high latency, downtime, crash or long-term unresponsiveness of the server is reduced.

[0093] In some preferred embodiments, in S500, the method for storing the file to be stored in the load balancer with the lowest fragmentation intensity value is as follows: the server receives the file to be stored from the client, stores the file to be stored in the load balancer with the lowest fragmentation intensity value, and adds the file storage structure information corresponding to the file to be stored as a node in the linked list in the memory of the load balancer; that is, the storage structure information of the file to be stored is added as a new node to the linked list in the corresponding storage server.

[0094] The file storage structure information includes: the file checksum (CheckSum), the volume name (VolName) of the storage disk in the storage server, the volume size (VolSize), the volume ID (VolId), the last write time (LdateTime) or storage timestamp, the file name (FileName), the total file size (FileSize), the file metadata (FileAttr), and the file validity flag (Flag).

[0095] In some preferred embodiments, in step S500, after the file to be stored is stored in the load balancer server, the linked list information in the corresponding memory of the load balancer server is updated.

[0096] Please see Figure 2 This invention also provides a massive file storage device, which includes:

[0097] The first module 100 is used to construct a linked list of all file storage structure information in each storage server and cache it in the memory of the corresponding storage server.

[0098] The second module 200 is used to receive storage requests for files to be stored sent by one or more clients to the server.

[0099] The third module 300 is used to calculate the fragmentation intensity of each storage service server in turn in response to storage requests;

[0100] The fourth module 400 is used to select storage servers whose fragmentation intensity has reached a balanced level as load balancers.

[0101] The fifth module, 500, is used to store the files to be stored in the load balancer with the lowest fragmentation intensity value.

[0102] The specific implementation of this massive file storage device is basically the same as the specific implementation of the massive file storage method described above, and will not be repeated here.

[0103] This invention also provides an electronic device, comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned massive file storage method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0104] It should be noted that the electronic device provided by the present invention can be a server, which is connected to each client via a wireless network, and the server is connected to each storage server via a wired network or a wireless network.

[0105] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated, the electronic device comprising:

[0106] The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0107] The memory 302 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random-access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301 using the massive file storage method of the embodiments of this invention.

[0108] Input / output interface 303 is used to implement information input and output;

[0109] The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0110] Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304);

[0111] The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.

[0112] This invention also provides a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described massive file storage method.

[0113] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0115] Those skilled in the art will understand that the technical solutions in the accompanying drawings do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than illustrated, or combine certain steps, or different steps.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0118] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0119] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0120] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A mass file storage method, characterized by, The method comprises the following steps: S100, all file storage structure information in each storage server is constructed into a linked list and cached into the memory of the corresponding each storage server; S200, a storage request of a file to be stored is received from one or more clients to the server end; S300, the fragmentation strength of each storage server is calculated in sequence in response to the storage request; S400, a storage server with balanced fragmentation strength is screened out as a balanced server; S500, the file to be stored is stored in the balanced server with the smallest fragmentation strength value; In S300, the method for calculating the fragmentation strength of each storage server in sequence is: The number of storage servers is SeCN, the access frequency of each node of the corresponding linked list in each storage server is Ut, the access frequency Ut is any one of the access times, query times, modified times or downloaded times of the file storage structure information of the node in the storage server; i1 represents the serial number of the storage server, i1∈[1, SeCN]; i1 is traversed in the range of [1, SeCN], and the centralized access trend value CE(i1) of the i1th storage server is calculated: ； Wherein, i2 is a variable, N(i1) is the number of nodes in the linked list corresponding to the i1th storage server, MaxUt(i1, i2) is the maximum value of the access frequency of the nodes from the 1st to the i2th in the linked list corresponding to the i1th storage server, MeanUt(i1, i2) is the mean value of the access frequency of the nodes from the 1st to the i2th in the linked list corresponding to the i1th storage server; the calculation method of the fragmentation strength of the i1th storage server is that the centralized access trend value CE(i1) of the i1th storage server is taken as the fragmentation strength of the i1th storage server; In S400, the method for screening out a storage server with balanced fragmentation strength as a balanced server is: Mark the storage servers with fragmentation strength greater than or equal to StrMean as unbalanced servers, and mark the storage servers with fragmentation strength less than StrMean as balanced servers; screen out the storage servers with fragmentation strength less than StrMean as balanced servers.

2. The mass file storage method according to claim 1, wherein, The method further comprises the step of S600: screening out the storage servers with unbalanced fragmentation strength as unbalanced servers, and performing a fragmentation synchronization operation or a disk fragmentation arrangement on each unbalanced server; wherein the fragmentation synchronization operation is to form a cluster by all unbalanced servers, and to perform load balancing on the files stored in the cluster by the load balancing method of FastDFS.

3. The mass file storage method of claim 1, wherein In S100, the all file storage structure information in each storage server is constructed into a linked list and cached into the memory of the corresponding each storage server, comprising: In each storage server, each file storage structure information is taken as a node, and a storage structure obtained by sequentially connecting each node in the order of the number of times each node is downloaded in the storage server from large to small is taken as a linked list, and then the linked list is cached in the memory of the corresponding each storage server.

4. The mass file storage method of claim 1, wherein, In S500, the method for storing the file to be stored into the balanced server with the smallest fragmentation intensity value is: The server end receives the file to be stored from the client end, stores the file to be stored into the balanced server with the smallest fragmentation intensity value, and adds the file storage structure information corresponding to the file to be stored as a node of the linked list into the linked list in the memory of the balanced server.

5. A mass file storage method, characterized by, The method comprises the following steps: S100, all file storage structure information in each storage server is constructed into a linked list and cached in the memory of the corresponding each storage server; S200, a storage request of a file to be stored is received from one or more client ends to the server end; S300, in response to the storage request, fragmentation intensity of each storage server is sequentially calculated; S400, a storage server with balanced fragmentation intensity is screened out as a balanced server; S500, the file to be stored is stored into the balanced server with the smallest fragmentation intensity value; In S300, the method for sequentially calculating the fragmentation intensity of each storage server is: The number of storage servers is denoted as SeCN, the access frequency of each node of the corresponding linked list in each storage server is denoted as Ut, the access frequency Ut is any one of the access times, the query times, the modification times or the download times of the file storage structure information of the node in the storage server, and the serial number of the storage server is denoted as i1; i1 is traversed in the range of [1, SeCN], and the centralized access trend value CE(i1) of the i1th storage server is calculated: ； wherein i2 is a variable, N(i1) is the number of nodes in the linked list corresponding to the ith1 storage server, is the access frequency of the ith2 node in the linked list corresponding to the ith1 storage server; MaxUt(i1,i2) is the maximum value in the access frequencies from the 1st to the ith2 node in the linked list corresponding to the ith1 storage server; MinUt(i1,i2) is the minimum value in the access frequencies from the 1st to the ith2 node in the linked list corresponding to the ith1 storage server; and log is a logarithmic function. The calculation method of the fragmentation intensity of the i1th storage server is that the centralized access trend value CE(i1) of the i1th storage server is taken as the fragmentation intensity of the i1th storage server; In S400, the method for screening out a storage server with balanced fragmentation intensity as a balanced server is: The average value of the fragmentation intensity of all storage servers is denoted as StrMean, the storage server with the fragmentation intensity greater than or equal to StrMean is marked as an unbalanced server, the storage server with the fragmentation intensity less than StrMean is marked as a balanced server, and the storage server with the fragmentation intensity less than StrMean is screened out as a balanced server.

6. A mass file storage device, characterized by comprising: The mass file storage method based on any one of claims 1 to 5, the device comprises: A first module is configured to construct all file storage structure information in each storage server into a linked list and cache the linked list in the memory of the corresponding each storage server; A second module is configured to receive a storage request of a file to be stored from one or more client ends to the server end; A third module is configured to sequentially calculate the fragmentation intensity of each storage server in response to the storage request; A fourth module is configured to screen out a storage server with balanced fragmentation intensity as a balanced server. The fourth module is used for screening the storage server with balanced fragmentation intensity as a balanced server; The fifth module is used for storing the file to be stored into the balanced server with the minimum fragmentation intensity.

7. An electronic device, comprising: The electronic device comprises a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the steps of the method according to any one of claims 1 to 5.

8. A computer readable storage medium for computer readable storage, characterized in that, The storage medium stores one or more programs, and the one or more programs are executable by one or more processors to realize the steps of the method according to any one of claims 1 to 5.

Images