File storage method and device and computing equipment

Abstract

The invention discloses a file storage method and device and computing equipment, and the method comprises the steps: receiving a first file requested to be stored by a requester; if the size of the first file does not reach the preset file size, constructing a first file chain table used for linking the first file, and writing the first file chain table into a merging buffer area; when it is monitored that the merged storage condition is met, a second file linked by a second file linked list in the merged buffer area is obtained; and writing the second file into the kernel cache, and refreshing the second file in the kernel cache into the storage device. By means of the mode, memory application and copying can be reduced, memory fragments can be reduced, expenditure caused by data copying can be reduced, processing resources can be saved, file storage efficiency can be improved, and storage performance of a system can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a file storage method, device, computing equipment, computer storage medium and computer program product. Background Art

[0002] File and data storage is one of the foundations of modern information technology and is vital to individuals, businesses and organizations. Object storage systems manage files in the form of objects to provide high-performance reading and writing. When storing files in this system, a fixed position offset is first allocated through metadata management to write to the corresponding memory location, and then the file is refreshed to a stable storage device through a write operation.

[0003] In actual applications, there will be a large number of files that need to be stored, including large files and small files. If each small file is stored separately, there will be problems such as increased file storage time and fragmentation of storage device space. Specifically: First, when there are a large number of small files, the seek time will increase, resulting in a longer file storage time; second, when writing files to the storage device, it is necessary to send an IO request (input and output request) to the storage device. Each time a small file is written, an IO request needs to be sent. When there are a large number of small files, the IO request will take a long time; in addition, writing small files directly to the storage device will cause the storage space of the storage device to be fragmented, affecting the overall performance of the storage device.

[0004] To solve the above problem, in some implementations, multiple small files are merged into a large file and then stored. During the storage process, the small files need to be copied to the memory, which will cause memory waste, increase data copy overhead and increase memory fragmentation. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide a file storage method, apparatus, computing device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0006] According to one aspect of the present application, a file storage method is provided, comprising:

[0007] Receiving a first file requested to be stored by a requesting party;

[0008] If the size of the first file does not reach the preset file size, construct a first file linked list for linking the first file, and write the first file linked list into the merge buffer;

[0009] When it is detected that the merge storage condition is met, the second file linked to the second file linked list in the merge buffer is obtained;

[0010] The second file is written into the kernel cache, and the second file in the kernel cache is refreshed into the storage device.

[0011] Optionally, before obtaining the second file linked to the second file linked list in the merge buffer, the method further includes:

[0012] When it is monitored that the total number of second file linked lists in the merge buffer reaches a first preset value and the cumulative sum of the sizes of the second files linked to the second file linked lists reaches a second preset value, it is determined that the merge storage condition is met;

[0013] When it is monitored that the total number of the second file linked lists in the merge buffer reaches a third preset value, it is determined that the merge storage condition is met, and the third preset value is greater than the first preset value;

[0014] When it is monitored that the accumulated sum of the sizes of the second files linked to the second file linked list in the merge buffer reaches a fourth preset value, it is determined that the merge storage condition is met, and the fourth preset value is greater than the second preset value;

[0015] When it is detected that the preset timer is triggered, it is determined that the merging storage condition is met.

[0016] Optionally, constructing a first file linked list for linking the first file further includes:

[0017] The header information corresponding to the first file is obtained, a linked list identifier is determined according to the header information, and pointer information corresponding to each data block in the first file is linked to the linked list identifier to obtain a first file linked list.

[0018] Optionally, after receiving the first file requested to be stored by the requesting party, the method further includes:

[0019] Obtaining pointer information of each file data block contained in the first file;

[0020] Sending pointer information of each file data block to the data transmission channel;

[0021] Before the step of linking the pointer information corresponding to each data block in the first file to the linked list identifier respectively, the method further comprises:

[0022] The pointer information corresponding to each data block in the first file is read from the data transmission channel.

[0023] Optionally, the method further comprises:

[0024] If the second file is successfully refreshed to the storage device, a notification of successful storage of the second file is returned to the requesting party corresponding to the second file;

[0025] If the second file is not successfully refreshed to the storage device, a notification of a storage failure of the second file is returned to the requesting party corresponding to the second file.

[0026] Optionally, obtaining the second file linked to the second file linked list in the merge buffer further includes:

[0027] Sending the second file linked list in the merge buffer to the data processing program;

[0028] Using a data processing program to access the second file link list to obtain the second file linked to the second file link list;

[0029] The method further includes: after sending the second file linked list in the merge buffer to the data processing program, clearing the merge buffer.

[0030] Optionally, writing the second file to the kernel cache further comprises:

[0031] The storage location information is calculated by accumulating and calculating the size of the second file through a data processing program, and the second file is written into the storage location indicated by the storage location information in the kernel cache.

[0032] Optionally, after receiving the first file requested to be stored by the requesting party, the method further includes:

[0033] If the size of the first file reaches the preset file size, the first file is written into the kernel cache, the first file in the memory cache is refreshed to the storage device, and a notification of successful storage of the first file is returned to the requesting party.

[0034] According to another aspect of the present application, there is provided a file storage device, comprising:

[0035] A receiving module, used for receiving a first file requested to be stored by a requesting party;

[0036] A first processing module, configured to construct a first file linked list for linking the first file if the size of the first file does not reach a preset file size, and write the first file linked list into a merge buffer;

[0037] The second processing module is used to obtain the second file linked to the second file linked list in the merge buffer when monitoring that the merge storage condition is met; write the second file into the kernel cache, and refresh the second file in the kernel cache to the storage device.

[0038] Optionally, the second processing module is further used for:

[0039] When it is monitored that the total number of second file linked lists in the merge buffer reaches a first preset value and the cumulative sum of the sizes of the second files linked to the second file linked lists reaches a second preset value, it is determined that the merge storage condition is met;

[0040] When it is monitored that the total number of the second file linked lists in the merge buffer reaches a third preset value, it is determined that the merge storage condition is met, and the third preset value is greater than the first preset value;

[0041] When it is monitored that the accumulated sum of the sizes of the second files linked to the second file linked list in the merge buffer reaches a fourth preset value, it is determined that the merge storage condition is met, and the fourth preset value is greater than the second preset value;

[0042] When it is detected that the preset timer is triggered, it is determined that the merging storage condition is met.

[0043] Optionally, the first processing module is further used for:

[0044] The header information corresponding to the first file is obtained, a linked list identifier is determined according to the header information, and pointer information corresponding to each data block in the first file is linked to the linked list identifier to obtain a first file linked list.

[0045] Optionally, the first processing module is further used for:

[0046] Obtaining pointer information of each file data block contained in the first file;

[0047] Sending pointer information of each file data block to the data transmission channel;

[0048] The pointer information corresponding to each data block in the first file is read from the data transmission channel.

[0049] Optionally, the device further includes: a feedback module, which is used to return a notification of successful storage of the second file to the requesting party corresponding to the second file if the second file is successfully refreshed to the storage device; if the second file is not successfully refreshed to the storage device, return a notification of failed storage of the second file to the requesting party corresponding to the second file.

[0050] Optionally, the second processing module is further used for:

[0051] Sending the second file linked list in the merge buffer to the data processing program;

[0052] Using a data processing program to access the second file link list to obtain the second file linked to the second file link list;

[0053] The device also includes: a cleaning module, which is used to clean up the merge buffer after sending the second file linked list in the merge buffer to the data processing program.

[0054] Optionally, the second processing module is further used for:

[0055] The storage location information is calculated by accumulating and calculating the size of the second file through a data processing program, and the second file is written into the storage location indicated by the storage location information in the kernel cache.

[0056] Optionally, the second processing module is further used to: if the size of the first file reaches a preset file size, write the first file into the kernel cache, and refresh the first file in the memory cache into the storage device;

[0057] The feedback module is further used to: return a notification of successful storage of the first file to the requesting party.

[0058] According to another aspect of the present application, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0059] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned file storage method.

[0060] According to another aspect of the present application, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned file storage method.

[0061] According to another aspect of the present application, a computer program product is provided, comprising at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned file storage method.

[0062] According to the file storage method, apparatus, computing device, computer storage medium and computer program product provided by the embodiments of the present application, the method includes: receiving a first file requested to be stored by a requesting party; if the size of the first file does not reach a preset file size, constructing a first file linked list for linking the first file, and writing the first file linked list into a merge buffer; when it is detected that the merge storage condition is met, obtaining the second file linked to the second file linked list in the merge buffer; writing the second file into a kernel cache, and refreshing the second file in the kernel cache to a storage device. Through the above method, on the one hand, by constructing a file linked list of files to be stored, when it is necessary to write files to the kernel cache, the corresponding files are read according to the file linked list. There is no need to copy the file memory during the whole process, which can reduce memory application and copying, reduce memory fragmentation and reduce the overhead caused by data copying; on the other hand, by merging the buffer to collect the file linked list of small files, when the merge storage conditions are met, the file linked list in the merge buffer is accessed to read the corresponding files, the read files are first written to the kernel cache, and then the files in the kernel cache are refreshed to the storage device, which can achieve the effect of writing multiple small files to the storage device at one time without merging multiple small files into a large file for storage, which can save processing resources and improve file storage efficiency, thereby improving the storage performance of the system.

[0063] Furthermore, in the process of merging and storing, it is only necessary to apply for a storage location once according to the accumulated sizes of the files to be merged and stored, which can speed up the process of applying for a file storage location.

[0064] Furthermore, the number of files to be merged and stored and the cumulative sum of the file sizes are limited. When the received files are generally small, as many files as possible can be stored in one merge storage process, which can further reduce the number of addressing times and IO request times, and can further improve the performance of the storage system. At the same time, by setting timer monitoring conditions, the requester can avoid long waiting times in low concurrency scenarios, ensure that the requester's storage request is completed in a shorter time, and avoid the request response time being too long and affecting the requester's user experience.

[0065] Furthermore, under the premise of realizing the one-time writing of multiple small files to the storage device, the storage result can be returned to the requester of each small file, and the situation where the storage of some files fails and the storage failure notification is returned to all requesters will not occur. The user experience can be improved and the retry rate can be reduced. Moreover, the storage success result will only be returned after the file is actually written to the storage device. The situation where the result given to the requester due to abnormal shutdown etc. is that the storage is successful but the memory data is ultimately lost can be effectively avoided. Data security can be guaranteed, and the storage result given to the requester is consistent with the actual file storage result.

[0066] Furthermore, when it is monitored that the merge storage conditions are met, a separate thread is started to perform merge storage processing, and after the second file linked list in the merge buffer is sent to the thread, the merge buffer is cleaned up in time, which can avoid the situation where there is too much data in the merge buffer and improve the performance of processing high concurrency scenarios.

[0067] Furthermore, by using a data transmission channel to transmit pointer information, the use of locks in the process of building a file linked list can be reduced, thereby improving processing performance.

[0068] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0070] Figure 1 A flowchart of a file storage method provided by an embodiment of the present application is shown;

[0071] Figure 2 A flowchart of a file storage method provided by another embodiment of the present application is shown;

[0072] Figure 3 A schematic diagram showing a process flow of a file storage method in another embodiment of the present application is shown;

[0073] Figure 4 A schematic diagram showing a process flow of a file storage method in another embodiment of the present application is shown;

[0074] Figure 5 A functional structure diagram of a file storage device provided in an embodiment of the present application is shown;

[0075] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0076] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0077] First, the terms involved in one or more embodiments of the present application are explained.

[0078] Object storage is a technology for storing and managing data in an unstructured format called objects.

[0079] Offset, the offset of the data in the storage unit relative to the head position.

[0080] Fsync operation is used to write data from the kernel cache to a stable storage device to ensure data persistence.

[0081] A data transmission channel, or channel, is a pipe with a buffer area that ensures concurrency safety. One party writes data into it and the other party reads data from it.

[0082] Figure 1 A flowchart of a file storage method provided by an embodiment of the present application is shown. Figure 1 As shown, the method comprises the following steps:

[0083] Step S110: receiving a first file requested to be stored by a requesting party.

[0084] The storage system includes a stable storage device, such as a disk, and writing a file to the storage device indicates that the file is stored successfully. When the requester needs to store the first file in the storage system, the first file is uploaded through a specified interface. Accordingly, the storage system receives the first file uploaded by the requester and finally stores the first file in the storage device through a series of processing.

[0085] Step S120: If the size of the first file does not reach the preset file size, a first file linked list for linking the first file is constructed, and the first file linked list is written into the merge buffer.

[0086] The preset file size is a file size threshold for distinguishing large files from small files, and the preset file size can be set according to actual business needs. If the size of the first file is greater than or equal to the threshold, the first file is considered to be a large file, and conversely, if the size of the first file is less than the threshold, the first file is considered to be a small file.

[0087] If the first file is a small file, a first file linked list for linking the first file is constructed, wherein the corresponding first file can be obtained by accessing the first file linked list, and then the first file linked list is written into the merge buffer.

[0088] Step S130, when it is detected that the merge storage condition is met, the second file linked to the second file link list in the merge buffer is obtained.

[0089] In order to distinguish it from the first file, the file linked list in the merge buffer is called the second file linked list. According to steps S110 to S120, it can be known that the second file linked list in the merge buffer is constructed and added to the merge buffer when the second file is previously received.

[0090] When the merge storage condition is met, the merge storage process for the second file linked to the second file linked list in the merge buffer is triggered. First, the second file linked to the second file linked list is read by accessing the second file linked list in the merge buffer.

[0091] Step S140: write the second file into the kernel cache, and refresh the second file in the kernel cache into the storage device.

[0092] The second file linked to the second file linked list in the merge buffer is written into the kernel cache, and then the second file in the kernel cache is refreshed to the disk. If there are multiple second file linked lists in the merge buffer, multiple small files can be written into the storage device at one time.

[0093] To summarize, according to the file storage method provided by this embodiment, on the one hand, by constructing a file linked list of files to be stored, when it is necessary to write files to the kernel cache, the corresponding files are read according to the file linked list. During the whole process, there is no need to copy the file memory, which can reduce memory application and copying, reduce memory fragmentation, and reduce the overhead caused by data copying; on the other hand, by merging the buffer to collect the file linked list of small files, when the merge storage conditions are met, the file linked list in the merge buffer is accessed to read the corresponding files, the read files are first written to the kernel cache, and then the files in the kernel cache are refreshed to the storage device, which can achieve the effect of writing multiple small files to the storage device at one time without merging multiple small files into a large file for storage, which can save processing resources and improve file storage efficiency, thereby improving the storage performance of the system.

[0094] Figure 2 A flowchart of a file storage method provided by another embodiment of the present application is shown. Figure 2 As shown, the method comprises the following steps:

[0095] Step S210: receiving a first file requested to be stored by a requesting party.

[0096] When the requester needs to store the first file in the storage system, the first file is uploaded through the designated interface. Accordingly, the storage system receives the first file uploaded by the requester and finally stores the first file in the storage device through a series of processing.

[0097] In actual applications, there are scenarios where the requester uploads a large number of files of different sizes at high concurrency. The receiving side (i.e., the storage system side) caches all received files for subsequent processing. After receiving the file, the file size is first determined, and the file size is compared with the preset file size through the filter to distinguish whether the received file is a large file or a small file.

[0098] Step S220: If the size of the first file reaches a preset file size, the first file is written into the kernel cache, the first file in the memory cache is refreshed into the storage device, and a notification of successful storage of the first file is returned to the requesting party.

[0099] If the size of the first file is greater than or equal to the preset file size, the first file is a large file, which is storage device friendly and can be directly written into the storage device. Specifically, firstly, a storage location information (i.e., Offset information) is requested according to the size of the first file, and then the first file is written into the corresponding location in the kernel cache according to the storage location information, and then the first file in the kernel cache is refreshed to the storage device through a write operation (i.e., Fsync operation), thereby completing the storage of the first file, and then returning a notification indicating that the first file is successfully stored to the requesting party.

[0100] Step S230: If the size of the first file does not reach the preset file size, a first file linked list for linking the first file is constructed, and the first file linked list is written into the merge buffer.

[0101] If the size of the first file is smaller than the preset file size, the first file is a small file, a first file linked list for linking the first file is constructed, and it is determined whether a merge buffer exists at this time. If not, a merge buffer is applied for, and the first file linked list is written into the applied merge buffer; if it exists, the first file linked list is written into the existing merge buffer.

[0102] The first file linked list is used to link the first file and does not store specific data of the first file. Instead, the specific data of the first file is read by accessing the first file linked list.

[0103] Specifically, the implementation method of constructing a first file linked list for linking the first file is: obtain the header information corresponding to the first file, determine the linked list identifier according to the header information, and link the pointer information corresponding to each data block in the first file to the linked list identifier, thereby obtaining the first file linked list. Specifically, first apply for header information for the first file, the header information contains the information uploaded by the requester, create a file linked list with the header information as the header (link list identifier), obtain the pointer information of each data block in the first file, and link each pointer information to the linked list identifier, thereby obtaining the first file linked list corresponding to the first file. Among them, each data block of the first file can be determined according to the specified length so that the size of each data block remains consistent.

[0104] In an optional manner, after receiving the first file requested to be stored by the requesting party, the pointer information of each file data block contained in the first file is obtained; the pointer information of each file data block is sent to the data transmission channel; before the step of linking the pointer information corresponding to each data block in the first file to the linked list identifier, the method further includes: reading the pointer information corresponding to each data block in the first file from the data transmission channel. That is, after obtaining the pointer information of each file data block contained in the first file, the pointer information of each file data block is first written into the data transmission channel, and then the pointer information is read from the data transmission channel to form a file linked list, wherein the data block pointer information of multiple files can be written to the data transmission channel concurrently, and the pointer information is read from the data transmission channel one by one, and each time a pointer information is read from the data transmission channel, the read pointer information is linked to the corresponding linked list identifier.

[0105] In some implementations, a locking method is used to cope with high concurrency, that is, after the file list construction of one file (including the calculation and linking of pointer information, etc.) is completed, the file list construction of the next file is carried out. By using a data transmission channel to transmit pointer information, the use of locks in the process of building the file list can be reduced, avoiding the problem of affecting processing performance by building the file list in a locking manner.

[0106] Furthermore, every time a complete file linked list is obtained, it is determined whether there is a merge buffer at this time. If yes, the file linked list is written into the existing merge buffer. If no, a new merge buffer is requested.

[0107] Step S240, when it is detected that the merge storage condition is met, the second file linked list in the merge buffer is sent to the data processing program, the merge buffer is cleared, and the second file linked to the second file linked list is obtained by accessing the second file linked list using the data processing program.

[0108] The method of the embodiment of the present application monitors from three dimensions, and determines whether the combined storage conditions are met according to the monitoring results of the three dimensions, as follows:

[0109] When it is monitored that the total number of second file linked lists in the merge buffer reaches a first preset value and the cumulative size of the second files linked to the second file linked list reaches a second preset value, it is determined that the merge storage condition is met; when it is monitored that the total number of second file linked lists in the merge buffer reaches a third preset value, it is determined that the merge storage condition is met, and the third preset value is greater than the first preset value; when it is monitored that the cumulative size of the second files linked to the second file linked list in the merge buffer reaches a fourth preset value, it is determined that the merge storage condition is met, and the fourth preset value is greater than the second preset value; when it is monitored that the preset timer is triggered, it is determined that the merge storage condition is met.

[0110] It should be noted that the reason for setting the timer monitoring dimension is that in a low-concurrency scenario, the second file linked lists collected in the merge buffer are relatively small, and the trigger conditions of the file size dimension and the file quantity dimension are difficult to meet or even cannot be met, which will increase the waiting time of the requester. In order to avoid the requester waiting for a long time, the next step of processing will be automatically started after a certain time has passed, and the wait for receiving new files will no longer continue, ensuring that the requester's storage request is completed in a shorter time, avoiding the request response time being too long and affecting the requester's experience. It can be seen that when the merge storage conditions are met, the number of second file linked lists in the merge buffer may be one or more.

[0111] For example, assuming that the first preset value is 10, the second preset value is 10M, the third preset value is 20, and the fourth preset value is 20M, when a large number of files uploaded by the requesting party are relatively small, when the total number of the second file linked list reaches 10, if the cumulative size of the second files linked to the second file linked list exceeds 10M at this time, it is determined that the merged storage condition is met, and the merged storage processing of the second files linked to the second file linked list in the merge buffer is triggered; conversely, if the cumulative size of the second files linked to the second file linked list does not exceed 10M at this time, it is determined that the merged storage condition is not met, and the process continues to wait for receiving new files to be stored; in the subsequent processing process, if the total number of the second file linked list reaches 20 or the cumulative size of the second files linked to the second file linked list reaches 20M or the preset timer is triggered, it is determined that the merged storage condition is met, and the merged storage processing of the second files linked to the second file linked list in the merge buffer is triggered.

[0112] Through the above method, the number of multiple files to be merged and stored and the cumulative sum of the file sizes are limited. When the received files are generally small, as many files as possible can be stored in one merge storage process, which can further reduce the number of addressing times and the number of IO requests, and can further improve the performance of the storage system. At the same time, by monitoring the triggering of the timer, it can avoid the request response time being too long to affect the requester's user experience.

[0113] The data processing program can be a thread, that is, when it is detected that the merge storage condition is met, a separate thread is started to perform merge storage processing, and after sending the second file list in the merge buffer to the thread, the merge buffer is cleaned up in time so that the file list of small files can be collected through the merge buffer later. In this way, the situation of too much data in the merge buffer can be avoided, and the performance of processing high-concurrency scenarios can be improved.

[0114] The data processing program is used to access the second file linked list to obtain the second file. As mentioned above, the second file linked list does not store the second file data, but stores the pointer information corresponding to each data block of the second file. The data processing program reads the corresponding data block from the second file through the pointer information, thereby obtaining all the data blocks in the second file.

[0115] Step S250, by means of a data processing program, the storage location information of the second file is accumulated and calculated according to the size of the second file, and the second file is written to the storage location indicated by the storage location information in the kernel cache; and the second file in the kernel cache is refreshed to the storage device.

[0116] The data processing program is also used to determine the storage location information (ie, offset information) of the second file, specifically by accumulating the size of the second file to be written into the kernel cache and applying for the storage location information, and then writing the second file into the corresponding location in the kernel cache.

[0117] After writing the second file to the kernel cache, the data processing program refreshes the second file in the kernel cache to the storage device through the Fsync operation. It can be seen that when there are multiple second file linked lists in the merge buffer, for the storage device, one large file is received at one time, and the large file contains multiple small files.

[0118] In an optional manner, after receiving the first file requested to be stored by the requesting party, the method further includes: calculating a check data block for each data block of the first file, wherein the check data block may be a CRC32 check data block calculated using a CRC32 algorithm, and the check data block is used to verify the correctness of the data.

[0119] It can be seen that through the embodiment of the present application, storing multiple second files only requires one IO request and one seek, and multiple second files can be written to the storage device at one time, which greatly reduces the IO request time and seek time, reduces the number of IO switching and scheduling, and is conducive to making full use of the performance of the storage device. It can avoid the problem of storage space fragmentation caused by writing small files separately, and greatly improves the system storage performance.

[0120] Step S260: Return the storage result of the second file to the requesting party corresponding to the second file.

[0121] Specifically, if the second file is successfully refreshed to the storage device, a notification of successful storage of the second file is returned to the requesting party corresponding to the second file; if the second file is not successfully refreshed to the storage device, a notification of failed storage of the second file is returned to the requesting party corresponding to the second file.

[0122] Before the second file is written to the storage device, the storage request of the requesting party is always in a suspended state. After the second file is successfully refreshed to the storage device, a notification indicating that the second file is successfully stored is returned to the requesting party of the second file. Correspondingly, if the second file is not successfully refreshed to the storage device, a notification indicating that the second file is failed to be stored is returned to the requesting party of the second file.

[0123] In this way, the storage results of multiple small files that are merged and stored do not affect each other. On the premise that multiple small files are written to the storage device at one time, the storage results can be returned to the requester of each small file, and the situation where the storage of some files fails and the storage failure notification is returned to all requesters will not occur, which can improve the user experience and reduce the retry rate. In addition, the storage success result will only be returned after the file is actually written to the storage device, which can effectively avoid the situation where the result given to the requester is that the storage is successful but the final memory data is lost due to abnormal shutdown and other reasons, which can ensure data security and ensure that the storage result given to the requester is consistent with the actual file storage result.

[0124] In an optional manner, a notification of successful storage of the second file is returned to the requester through the data transmission channel on the requester side of the second file. Further, in response to the notification of successful storage of the second file, the data transmission channel on the requester side of the second file is destroyed. The data transmission channel on the requester side will be destroyed when not needed, ensuring that the data transmission channel on the requester side will not grow indefinitely. At the same time, before receiving the file storage result, the data transmission channel on the requester side will wait, thereby ensuring that each storage request is valid. Combined with the monitoring method of regular inspection, it is guaranteed that the requester side can receive the file storage result.

[0125] In summary, according to the file storage method provided by this embodiment, on the one hand, by constructing a file linked list of files to be stored, when it is necessary to write files to the kernel cache, the corresponding files are read according to the file linked list, and there is no need to perform file memory copying during the whole process, which can reduce memory application and copying, and can reduce memory fragmentation and reduce the overhead caused by data copying; on the other hand, by merging the buffer to collect the file linked list of small files, when the merge storage condition is met, the file linked list in the merge buffer is accessed to read the corresponding file, and the read file is first written to the kernel cache, and then the file in the kernel cache is refreshed to the storage device, which can achieve the goal of writing multiple small files at one time. On the one hand, the effect of storing multiple small files in a storage device can be achieved without merging multiple small files into a large file for storage, which can save processing resources, improve file storage efficiency, and improve system storage performance; on the other hand, in the process of merging and storing, it is only necessary to apply for a storage location once according to the cumulative size of the files to be merged and stored, which can speed up the application process of the file storage location; on the other hand, when it is detected that the merge storage conditions are met, a separate thread is started for merge storage processing, and after the second file linked list in the merge buffer is sent to the thread, the merge buffer is cleaned up in time, which can avoid the situation where there is too much data in the merge buffer and can also improve the processing of high concurrency scenarios On the other hand, by transmitting pointer information through the data transmission channel, the use of locks in the process of building the file list can be reduced, thereby improving processing performance; on the other hand, the number of multiple files to be merged and stored and the cumulative sum of the file sizes are limited. In the case where the received files are generally small, as many files as possible can be stored in a single merge storage process, which can further reduce the number of addressing times and the number of IO requests, and can further improve the performance of the storage system. At the same time, by setting a timer monitoring condition, the requester can avoid waiting for a long time in a low-concurrency scenario, ensuring that the requester's storage request is completed in a shorter time, and avoiding the request response time being too long to affect the requester's use On the other hand, the storage results of multiple small files that are merged and stored do not affect each other. On the premise that multiple small files are written to the storage device at one time, the storage result can be returned to the requester of each small file, and there will be no situation where the storage of some files fails and the storage failure notification is returned to all requesters. This can improve the user experience and reduce the retry rate. Moreover, the storage success result will only be returned after the file is actually written to the storage device. This can effectively avoid the situation where the result for the requester is that the storage is successful but the final memory data is lost due to reasons such as abnormal shutdown. This can ensure data security and ensure that the storage result for the requester is consistent with the actual file storage result.

[0126] Figure 3 A schematic diagram of a process flow of a file storage method in another embodiment of the present application is shown. As shown in the figure, the process flow of the file storage method is as follows:

[0127] The received set of files to be stored 1 includes multiple files of different sizes and formats. The large files and small files are distinguished by the filter 2, and the large files are divided into the large file queue 3, and the small files are divided into the small file queue 4. For the large files, they are directly written into the kernel cache 6, and then the large files in the kernel cache 6 are written into the disk 7 through the disk write operation. For each small file, a file linked list is constructed and written into the merge buffer, and the size detector 41 (used to monitor the sum of the sizes of the files linked by all the file linked lists in the merge buffer), the timer 42 and the number detector (not shown in the figure, used to monitor the total number of all the file linked lists in the merge buffer) are used for monitoring. When the merge storage condition is detected, the files linked by each file linked list in the merge buffer are read respectively and written into the kernel cache 6, so that the effect of writing the merged file 5 into the kernel cache 6 can be achieved. The merged file 5 is essentially composed of multiple independent small files. Finally, the merged file 5 in the kernel cache 6 is written into the disk 7 through the disk write operation.

[0128] Figure 4 A schematic diagram of a process flow of a file storage method in another embodiment of the present application is shown. As shown in the figure, the method specifically includes the following steps:

[0129] Step S401, receiving a file;

[0130] Step S402, distinguish between large files and small files; if it is a large file, execute steps S403 and S404. If it is a small file, execute step S405.

[0131] Step S403, writing into kernel cache;

[0132] Step S404, writing to disk;

[0133] That is, if the received file is a large file, it is directly written to the kernel cache and then written to the disk through a disk write operation.

[0134] Step S405, sending pointer information of the data block through the data transmission channel;

[0135] Step S406, constructing a file linked list of small files;

[0136] Step S407, check whether there is a merge buffer; that is, every time the file linked list is constructed, determine whether there is a merge buffer. If so, execute step S409, if not, execute steps S408 and S409 in sequence.

[0137] Step S408, applying for merging buffers;

[0138] Step S409, the file list is written into the merge buffer;

[0139] Step S410, monitoring timer;

[0140] Step S411, monitoring the total size of the files linked to the file linked list in the merge buffer;

[0141] Step S412, monitoring the total number of file linked lists in the merge buffer.

[0142] Step S413, judging whether the combined storage condition is met according to the monitoring result;

[0143] Step S414, start the data processing program;

[0144] Step S415, reclaiming the merge buffer;

[0145] Step S416, waiting for receiving a new file.

[0146] Among them, step S410, step S411, and step S412 are executed in parallel, and it is determined whether the merge storage condition is met according to the monitoring result. For the specific implementation method, please refer to the description in the above embodiment, which will not be repeated here. If the merge storage condition is met, step S414, step S403, and step S404 are executed in sequence, and the data processing program is started to write the files linked to each file list in the merge buffer into the kernel cache, and then write the files in the kernel cache to the disk, and step S415 is executed in parallel. If the merge storage condition is not met, step S416 is executed.

[0147] Figure 5 FIG. 1 shows a functional structure diagram of a file storage device provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0148] A receiving module 51, configured to receive a first file requested to be stored by a requesting party;

[0149] A first processing module 52 is used for constructing a first file link list for linking the first file if the size of the first file does not reach a preset file size, and writing the first file link list into a merge buffer;

[0150] The second processing module 53 is used to obtain the second file linked to the second file linked list in the merge buffer when monitoring that the merge storage condition is met; write the second file into the kernel cache, and refresh the second file in the kernel cache to the storage device.

[0151] In an optional manner, the second processing module 53 is further used for:

[0152] When it is monitored that the total number of second file linked lists in the merge buffer reaches a first preset value and the cumulative sum of the sizes of the second files linked to the second file linked lists reaches a second preset value, it is determined that the merge storage condition is met;

[0153] When it is monitored that the total number of the second file linked lists in the merge buffer reaches a third preset value, it is determined that the merge storage condition is met, and the third preset value is greater than the first preset value;

[0154] When it is monitored that the accumulated sum of the sizes of the second files linked to the second file linked list in the merge buffer reaches a fourth preset value, it is determined that the merge storage condition is met, and the fourth preset value is greater than the second preset value;

[0155] When it is detected that the preset timer is triggered, it is determined that the merging storage condition is met.

[0156] In an optional manner, the first processing module 52 is further configured to:

[0157] The header information corresponding to the first file is obtained, a linked list identifier is determined according to the header information, and pointer information corresponding to each data block in the first file is linked to the linked list identifier to obtain a first file linked list.

[0158] In an optional manner, the first processing module 52 is further configured to:

[0159] Obtaining pointer information of each file data block contained in the first file;

[0160] Sending pointer information of each file data block to the data transmission channel;

[0161] The pointer information corresponding to each data block in the first file is read from the data transmission channel.

[0162] In an optional manner, the device further includes: a feedback module, which is used to return a notification of successful storage of the second file to the requesting party corresponding to the second file if the second file is successfully refreshed to the storage device; if the second file is not successfully refreshed to the storage device, return a notification of failed storage of the second file to the requesting party corresponding to the second file.

[0163] In an optional manner, the second processing module 53 is further used for:

[0164] Sending the second file linked list in the merge buffer to the data processing program;

[0165] Using a data processing program to access the second file link list to obtain the second file linked to the second file link list;

[0166] The device also includes: a cleaning module, which is used to clean up the merge buffer after sending the second file linked list in the merge buffer to the data processing program.

[0167] In an optional manner, the second processing module 53 is further used for:

[0168] The storage location information is calculated by accumulating and calculating the size of the second file through a data processing program, and the second file is written into the storage location indicated by the storage location information in the kernel cache.

[0169] In an optional manner, the second processing module 53 is further used to: if the size of the first file reaches a preset file size, write the first file into the kernel cache, and refresh the first file in the memory cache to the storage device;

[0170] The feedback module is further used to: return a notification of successful storage of the first file to the requesting party.

[0171] To summarize, according to the file storage device provided by this embodiment, on the one hand, by constructing a file linked list of files to be stored, when it is necessary to write files to the kernel cache, the corresponding files are read according to the file linked list. During the whole process, there is no need to perform file memory copying, which can reduce memory application and copying, reduce memory fragmentation, and reduce the overhead caused by data copying; on the other hand, by merging the buffer to collect the file linked list of small files, when the merge storage conditions are met, the file linked list in the merge buffer is accessed to read the corresponding files, the read files are first written to the kernel cache, and then the files in the kernel cache are refreshed to the storage device, which can achieve the effect of writing multiple small files to the storage device at one time without merging multiple small files into a large file for storage, which can save processing resources and improve file storage efficiency, thereby improving the storage performance of the system.

[0172] An embodiment of the present application provides a non-volatile computer storage medium, which stores at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the file storage method in any of the above method embodiments.

[0173] An embodiment of the present application provides a computer program product, which includes at least one executable instruction or computer program, and the executable instruction or computer program can enable a processor to perform operations corresponding to the file storage method in any of the above method embodiments.

[0174] Figure 6 A schematic diagram of the structure of an embodiment of a computing device of the present application is shown. The specific embodiment of the present application does not limit the specific implementation of the computing device.

[0175] like Figure 6As shown, the computing device may include: a processor (processor) 602 , a communications interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0176] The processor 602, the communication interface 604, and the memory 606 communicate with each other via a communication bus 608. The communication interface 604 is used to communicate with other devices such as a client or other server network elements. The processor 602 is used to execute a program 610, which can specifically execute the relevant steps in the above-mentioned file storage method embodiment for a computing device.

[0177] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0178] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0179] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one storage device memory.

[0180] The program 610 can be specifically used to enable the processor 602 to execute the file storage method in any of the above method embodiments. The specific implementation of each step in the program 610 can refer to the corresponding description of the corresponding steps and units in the file storage embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, which will not be repeated here.

[0181] The algorithm or display provided here are not inherently related to any specific computer, virtual system or other equipment. Various general systems can also be used together with the teaching based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the present application embodiment is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present application described here, and the above description of specific languages ​​is to disclose the best mode of implementation of the present application.

[0182] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0183] Similarly, it should be understood that in order to streamline the present application and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the embodiments of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0184] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0185] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0186] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components according to the embodiments of the present application. The application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing a part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0187] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be constructed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A file storage method, comprising: Receiving a first file requested to be stored by a requesting party; If the size of the first file does not reach the preset file size, construct a first file linked list for linking the first file, and write the first file linked list into a merge buffer; When it is detected that the merge storage condition is met, obtaining the second file linked to the second file linked list in the merge buffer; The second file is written into the kernel cache, and the second file in the kernel cache is refreshed into the storage device.

2. The method according to claim 1, wherein: Before obtaining the second file linked to the second file linked list in the merge buffer, the method further includes: When it is monitored that the total number of second file linked lists in the merge buffer reaches a first preset value and the accumulated size of the second files linked to the second file linked lists reaches a second preset value, it is determined that the merge storage condition is met; When it is monitored that the total number of the second file linked lists in the merge buffer reaches a third preset value, it is determined that the merge storage condition is met, and the third preset value is greater than the first preset value; When it is monitored that the accumulated sum of the sizes of the second files linked to the second file linked list in the merge buffer reaches a fourth preset value, it is determined that the merge storage condition is met, and the fourth preset value is greater than the second preset value; When it is detected that the preset timer is triggered, it is determined that the merging storage condition is met.

3. The method according to claim 1 or 2, wherein: The constructing a first file linked list for linking the first file further comprises: The header information corresponding to the first file is obtained, a linked list identifier is determined according to the header information, and pointer information corresponding to each data block in the first file is linked to the linked list identifier to obtain the first file linked list.

4. The method according to claim 3, wherein: After receiving the first file requested to be stored by the requesting party, the method further comprises: Obtaining pointer information of each file data block contained in the first file; Sending pointer information of each file data block to the data transmission channel; Before the step of respectively linking the pointer information corresponding to each data block in the first file to the linked list identifier, the method further comprises: Pointer information corresponding to each data block in the first file is read from the data transmission channel.

5. The method according to claim 1 or 2, wherein: The method further comprises: If the second file is successfully refreshed to the storage device, a notification of successful storage of the second file is returned to the requesting party corresponding to the second file; If the second file is not successfully refreshed to the storage device, a notification of a storage failure of the second file is returned to the requesting party corresponding to the second file.

6. The method according to claim 1 or 2, wherein: The obtaining of the second file linked to the second file linked list in the merge buffer further comprises: Sending the second file linked list in the merge buffer to a data processing program; Using the data processing program to access the second file link list, obtain the second file linked to the second file link list; The method further includes: after sending the second file linked list in the merge buffer to the data processing program, clearing the merge buffer.

7. The method according to claim 6, wherein: Writing the second file into the kernel cache further comprises: The data processing program calculates storage location information based on the accumulated size of the second file, and writes the second file into a storage location indicated by the storage location information in the kernel cache.

8. The method according to claim 1 or 2, wherein: After receiving the first file requested to be stored by the requesting party, the method further comprises: If the size of the first file reaches a preset file size, the first file is written into the kernel cache, the first file in the memory cache is refreshed to the storage device, and a notification of successful storage of the first file is returned to the requesting party.

9. A file storage device, comprising: A receiving module, used for receiving a first file requested to be stored by a requesting party; A first processing module, configured to construct a first file linked list for linking the first file if the size of the first file does not reach a preset file size, and write the first file linked list into a merge buffer; A second processing module is used to obtain the second file linked to the second file linked list in the merge buffer when monitoring that the merge storage condition is met; The second file is written into the kernel cache, and the second file in the kernel cache is refreshed into the storage device.

10. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the file storage method according to any one of claims 1 to 8.

11. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and wherein the executable instruction enables a processor to execute an operation corresponding to the file storage method according to any one of claims 1 to 8.

12. A computer program product, comprising at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the file storage method according to any one of claims 1 to 8.

Images