File writing method and device, storage medium and electronic device

Abstract

An embodiment of the present invention discloses a file writing method, apparatus, storage medium, and electronic device. The method includes: receiving an application request sent by a file client, where the application request carries first attribute information corresponding to a first file; in response to the application request, determining a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and obtaining file information of the second file, where the second file is used for storage multiplexing with the first file; sending the file information of the second file to the file client, and instructing the file client to write the first file to a disk corresponding to the second file according to the file information.

Description

Technical Field

[0001] The present invention relates to the field of distributed block storage, and in particular, to a file writing method, a device, a storage medium, and an electronic device. Background Art

[0002] When storing in the security field, there are a large number of alarm videos and pictures to be stored. These files vary in size. The average size of an alarm video is 10MB, and a picture may only occupy 4K of space. For such small files, the common practice in the industry is to merge and store them.

[0003] In related technologies, small files are aggregated into large files and then written to the OSD at one time to solve the problem of frequent pressure on the system caused by frequent writing of small files. The implementation method is to first determine whether the small files included in the object data writing request are less than or equal to a preset value. If so, it is determined whether it is the first time to write a small file. If so, the data in the small file is written into the aggregated file, and the identifier of the aggregated file, the length of the data, and the position of the data in the aggregated file are stored in the metadata of the small file; it is determined whether the capacity of the aggregated file exceeds the set capacity. If so, the data is written into a new aggregated file. If not, the step of receiving the object writing request is returned; the aggregated file that reaches the set capacity is written into the OSD, and the metadata is stored in the database.

[0004] However, in related technologies, when merging small files, it is not possible to select a suitable file for merging according to the attribute information of the small files, resulting in a relatively large disk space occupation after merging.

[0005] In view of the related technologies, in the process of merging and storing small files by traditional methods, it is not possible to select a suitable file for merging and storing small files according to the attributes of the small files, resulting in a problem of low disk space utilization. At present, no effective solution has been proposed.

[0006] Therefore, it is necessary to improve the related technologies to overcome the above-mentioned defects in the related technologies. Summary of the Invention

[0007] Embodiments of the present invention provide a file writing method, a device, a storage medium, and an electronic device, so as to at least solve the problem that in the process of merging and storing small files by traditional methods, it is not possible to select a suitable file for merging and storing small files according to the attributes of the small files, resulting in low disk space utilization.

[0008] According to one aspect of an embodiment of the present invention, a file writing method is provided, including: receiving an application request sent by a file client, where the application request carries first attribute information corresponding to a first file; in response to the application request, determining a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and obtaining file information of the second file, where the second file is used for storage multiplexing with the first file; sending the file information of the second file to the file client, and instructing the file client to write the first file to a disk corresponding to the second file according to the file information.

[0009] Further, in response to the application request, determining a second file corresponding to second attribute information that matches the first attribute information in a metadata management server includes: obtaining a multiplexing file list saved in the metadata management server, where the multiplexing file list includes: multiple files used for storage multiplexing with the first file; determining whether there is a second file in the multiplexing file list that satisfies a first preset condition, where the first preset condition is used to indicate that the first attribute information matches the second attribute information;

[0010] Further, in the case of existence, determining the file that satisfies the first preset condition in the multiplexing file list as the second file; in the case of non-existence, creating the second file in the metadata management server.

[0011] Further, the first preset condition includes: the erasure code in the first attribute information is the same as the erasure code in the second attribute information; when the first file and the second file are in the same file object bucket, the file attributes in the first attribute information are the same as the file attributes in the second attribute information.

[0012] Further, creating the second file in the metadata management server includes: obtaining a Zone group multiplexing list saved in the metadata management server; determining that there is a Zone group in the Zone group multiplexing list that satisfies a second preset condition, and obtaining a data node group corresponding to the Zone group and a Zone group number of the Zone group; creating the second file according to the data node group corresponding to the Zone group and the Zone group number of the Zone group.

[0013] Further, after obtaining the Zone group reuse list stored in the metadata management server, the method further includes: determining that there is no Zone group in the Zone group reuse list that meets the second preset condition; instructing the load balancing module of the metadata management server to determine the data node group and the Zone group number, and creating the second file according to the data node group and the Zone group number determined by the load balancing module.

[0014] Further, the second preset condition includes: the second file and the Zone group in the Zone group reuse list have the same erasure code type; the files stored in the second file and the Zone group in the Zone group reuse list are located in the same object bucket; the Zone group in the Zone group reuse list is used to store the first file.

[0015] Further, instructing the file client to transfer the first file to the disk corresponding to the second file according to the file information includes: instructing the file client to divide the first file into multiple data blocks, and sending the multiple data blocks and the Zone group number in the file information to multiple data nodes corresponding to the data node group in the file information; controlling each data node in the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number.

[0016] Further, after controlling each data node in the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number according to the Zone group number, the method further includes: controlling the data node to send the metadata information of the corresponding data block and the metadata information of the Zone to the metadata management server, so that the metadata management server saves the metadata information of the data block and the metadata information of the Zone to the file object cache.

[0017] Further, after instructing the file client to write the first file to the disk corresponding to the second file according to the file information, the method further includes: when the second file meets the third preset condition, adding the second file to the reuse file list of the metadata management server; when the second file does not meet the third preset condition, determining whether the Zone group where the second file is located meets the fourth preset condition; when the Zone group meets the fourth preset condition, adding the Zone group to the Zone group reuse list of the metadata management server.

[0018] Further, the third preset condition includes: the second file is within the reuse cycle; each data block of the second file has the same length; the status of the second file is the normal closed status; the remaining space of the disk Zone corresponding to the second file is greater than the first preset threshold.

[0019] Further, the fourth preset condition includes: the Zone group can be aligned for writing; the data nodes of the Zone group are in the active state; the Zone group is within the reusable cycle; the remaining space of the Zone group is greater than the second preset threshold.

[0020] Further, the method further includes: when the reuse file list and the Zone group reuse list in the metadata management server change, saving the reuse file list and the Zone group reuse list to the log data, and sending the log data to the target metadata management server.

[0021] According to another aspect of the embodiments of the present invention, there is also provided a file writing device, including: a receiving module, configured to receive an application request sent by a file client, where the application request carries first attribute information corresponding to a first file; an obtaining module, configured to, in response to the application request, determine a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and obtain file information of the second file, where the second file is used for storage reuse with the first file; a writing module, configured to send the file information of the second file to the file client, and instruct the file client to write the first file to a disk corresponding to the second file according to the file information.

[0022] According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the above file writing method when running.

[0023] According to still another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the above processor executes the above file writing method through the computer program.

[0024] Through the present invention, an application request sent by a receiving file client is received, where the application request carries first attribute information corresponding to a first file; in response to the application request, a second file corresponding to second attribute information matching the first attribute information is determined in a metadata management server, and file information of the second file is obtained, where the second file is used for storage multiplexing with the first file; the file information of the second file is sent to the file client, and the file client is instructed to write the first file to a disk corresponding to the second file according to the file information. That is to say, the metadata management server can select a second file that allows multiplexing for the first file according to the attributes of the first file sent by the file client, and send the file information of the second file to the file client, and then the file client can write the first file to the disk corresponding to the second file according to the file information. By adopting the above technical solution, the problem that in the process of merging and storing small files in the traditional method, a suitable file cannot be selected for the small files according to the attributes of the small files, resulting in low disk space utilization, is solved. Furthermore, a suitable file is selected for the small files for merging and storing, improving the disk space utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention, and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a hardware structure block diagram of a computer terminal for a file writing method according to an embodiment of the present invention;

[0027] Figure 2 is a flowchart (1) of a file writing method according to an embodiment of the present invention;

[0028] Figure 3 is a flowchart (2) of a file writing method according to an embodiment of the present invention;

[0029] Figure 4 is a flowchart (3) of a file writing method according to an embodiment of the present invention;

[0030] Figure 5 is a server module diagram of a file writing method according to an embodiment of the present invention;

[0031] Figure 6 is a structure block diagram of a file writing device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances, so that the embodiments of the present invention described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] The method embodiments provided in the embodiments of the present application can be executed on a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal for the file writing method of the embodiments of the present invention. As Figure 1 shown, the computer terminal may include one or more ( Figure 1 only one is shown in the figure) processors 102 (the processor 102 may include but is not limited to a microprocessor (abbreviated as MPU) or a programmable logic device (abbreviated as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the above-mentioned computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may further include more or fewer components than those shown in Figure 1 , or have the same functions as those shown in Figure 1 or different configurations with more functions than those shown in Figure 1 .

[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the file writing method in the embodiments of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above-mentioned method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the computer terminal through a network. Examples of the above network include but are not limited to the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer terminal. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] In this embodiment, a file writing method is provided. Figure 2 It is a flowchart (one) of the file writing method according to the embodiments of the present invention, and the process includes the following steps:

[0038] Step S202, receiving an application request sent by a file client, where the application request carries first attribute information corresponding to a first file;

[0039] It should be noted that the space occupied by the first file is less than a preset threshold, and the preset threshold may be 100MB, and this embodiment does not make specific limitations.

[0040] Step S204, in response to the application request, determining a second file corresponding to second attribute information matching the first attribute information in the metadata management server, and obtaining file information of the second file, where the second file is used for storage multiplexing with the first file;

[0041] It should be noted that the file information includes: data node group, Zone group number.

[0042] Step S206: Send the file information of the second file to the file client, and instruct the file client to write the first file to the disk corresponding to the second file according to the file information.

[0043] Through the present invention, an application request sent by a file client is received, where the application request carries first attribute information corresponding to a first file; in response to the application request, a second file corresponding to second attribute information matching the first attribute information is determined in a metadata management server, and file information of the second file is obtained, where the second file is used for storage multiplexing with the first file; the file information of the second file is sent to the file client, and the file client is instructed to write the first file to the disk corresponding to the second file according to the file information. That is to say, the metadata management server can select a second file that allows multiplexing for the first file according to the attributes of the first file sent by the file client, and send the file information of the second file to the file client. Then, the file client can write the first file to the disk corresponding to the second file according to the file information. By adopting the above technical solution, the problem that in the process of merging and storing small files in the traditional method, a suitable file cannot be selected for the small files according to the attributes of the small files, resulting in low disk space utilization, is solved. Furthermore, a suitable file is selected for the small files for merging and storing, improving the disk space utilization.

[0044] It should be noted that there are multiple implementation manners for the above step S204. In an optional embodiment, it is implemented by the following method: Obtain a multiplexing file list saved in the metadata management server, where the multiplexing file list includes: multiple files used for storage multiplexing with the first file; determine whether there is a second file in the multiplexing file list that meets a first preset condition, where the first preset condition is used to indicate that the first attribute information matches the second attribute information; if it exists, determine the file that meets the first preset condition in the multiplexing file list as the second file; if it does not exist, create the second file in the metadata management server.

[0045] That is to say, a multiplexing file list is saved in the metadata management server. After the metadata management server obtains the first attribute information of the first file, it will search in the multiplexing file list to determine whether there is a reusable file that meets the first preset condition in the multiplexing file list. If there is, the reusable file is determined as the second file. If there is no reusable file that meets the first preset condition, the metadata management server will create a second file according to the first attribute information of the first file.

[0046] It should be noted that the first preset condition includes: the erasure code in the first attribute information is the same as the erasure code in the second attribute information; when the first file and the second file are in the same file object bucket, the file attributes in the first attribute information are the same as the file attributes in the second attribute information.

[0047] That is to say, the requirements for the second file that can be reused with the first file in the reused file list are: the EC erasure code types of the first file and the second file are the same; the first file and the second file are in the same file object bucket and have the same file attributes, where the file attributes include: lifecycle, readable permission.

[0048] To better understand the creation of the second file in the metadata management server, in an optional embodiment, it is implemented through the following technical solution: obtain the Zone group reuse list saved in the metadata management server; determine that there is a Zone group in the Zone group reuse list that meets the second preset condition, and obtain the data node group corresponding to the Zone group and the Zone group number of the Zone group; create the second file according to the data node group corresponding to the Zone group and the Zone group number of the Zone group.

[0049] Further, after obtaining the Zone group reuse list saved in the metadata management server, determine that there is no Zone group in the Zone group reuse list that meets the second preset condition; instruct the load balancing module of the metadata management server to determine the data node group and the Zone group number, and create the second file according to the data node group and the Zone group number determined by the load balancing module.

[0050] It should be noted that in the metadata management server, an erasure code will be first assigned to the newly created second file, and the assigned erasure code is the erasure code of the first file. Then, after obtaining the Zone reuse list, determine whether there is a Zone group in the Zone group reuse list that meets the second preset condition. If so, create the second file according to the data node group corresponding to the Zone group and the Zone group number of the Zone group. If there is no Zone group that meets the second preset condition, instruct the load balancing module of the metadata management server to determine a new data node group and a new Zone group number, and create the second file with the new data node group and the new Zone group number.

[0051] It should be noted that the second preset condition includes: the second file and the Zone group in the Zone group reuse list have the same erasure code type; the files stored in the second file and the Zone group in the Zone group reuse list are in the same object bucket; the Zone group in the Zone group reuse list is used to store the first file.

[0052] That is to say, the requirements for the reusable Zone groups in the Zone group reuse list are as follows: the erasure code type of the newly created second file is the same as that of the reusable Zone group; the newly created second file and the files stored in the reusable Zone group are located in the same object bucket; and the reusable Zone group is used to store the first file.

[0053] To better understand the above step S206, in an optional embodiment, it is implemented as follows: instruct the file client to divide the first file into multiple data blocks, and send the multiple data blocks and the Zone group number in the file information to multiple data nodes corresponding to the data node group in the file information; control each of the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number.

[0054] That is to say, the file information includes: a data node group, a Zone group number. It is indicated that after the small file client obtains the file information of the second file, the first file is divided into multiple data blocks and written to the corresponding data nodes in the file information in an aligned manner. When writing, the Zone group number in the file information needs to be sent to the data nodes. The data nodes will query the local cache according to the Zone group number to confirm whether the Zone group number already exists. If it does not exist, the data nodes will select a real disk according to the load balancing algorithm of the data nodes and write the data. After the data is successfully written, the information including the disk and the Zone group number information is refreshed to the metadata database of the data nodes for persistence. The database is usually RocksDB. If the data nodes confirm that the Zone group number exists, the data is appended and written to the Zone of the corresponding disk.

[0055] Further, after controlling each of the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number, it is necessary to control the data nodes to send the metadata information of the corresponding data block and the metadata information of the Zone to the metadata management server, so that the metadata management server saves the metadata information of the data block and the metadata information of the Zone to the file object cache.

[0056] It should be noted that after writing the first file to the disk, the file client needs to notify the data node and the metadata management server to close the file. After the file is closed, the data node reports the metadata information of the data block and the metadata information of the Zone to the metadata management server. The metadata information of the Zone includes the Zone group number, the total length of the Zone, the used length of the Zone, and the creation time of the Zone. The metadata management server can determine that the writing of the Zone group is completed only after all the Zone information on all data nodes under the Zone group number is reported. At the same time, the second file also needs to wait until all the metadata information of all data blocks is reported before it can be considered that the first I / O write of the second file is completed.

[0057] Further, after instructing the file client to write the first file to the disk corresponding to the second file according to the file information, the method further includes: when the second file meets the third preset condition, adding the second file to the reusable file list of the metadata management server; when the second file does not meet the third preset condition, determining whether the Zone group where the second file is located meets the fourth preset condition; when the Zone group meets the fourth preset condition, adding the Zone group to the Zone group reusable list of the metadata management server.

[0058] It should be noted that in order to better save disk space, the metadata management server can judge whether the second file can be reused again according to the third preset condition. If the second file can be reused, the second file is added to the reusable file list; when the large file cannot be reused, then judge whether the Zone group where the second file is located can be reused. If the Zone group meets the fourth preset condition, the Zone group is added to the Zone group reusable list.

[0059] It should be noted that the third preset condition includes: the second file is within the reuse cycle; the length of each data block of the second file is equal; the status of the second file is the normal closed state; the remaining space of the disk Zone corresponding to the second file is greater than the first preset threshold. The fourth preset condition includes: the Zone group can be written in alignment; the data nodes of the Zone group are in the active state; the Zone group is within the reusable cycle; the remaining space of the Zone group is greater than the second preset threshold.

[0060] That is to say, the third preset condition for the second file to be reusable is as follows: within the reuse cycle of the second file, where the reuse cycle of the second file can be set to 1 day to avoid too long a time span between the earliest written first file and the latest written first file in the same second file, and this time cycle is default configurable; the length of each data block of the second file is equal to ensure that all written data blocks are aligned; the second file switches from the writing state to the normal closed state to avoid the large file being allocated for reuse during the writing process; the remaining space of the SMR disk Zone corresponding to the large file is greater than the first preset threshold, where the first preset threshold can be set to 100 MB to ensure that at least one first file can be completely written.

[0061] The fourth preset condition that the Zone group needs to meet to be added to the reuse list is as follows: the Zone group can be written in alignment; all data nodes of the Zone group are online and valid; within the reusable cycle of the Zone group, where the default reusable cycle time of the Zone group can be set to 7 days; there is still remaining writable space in the Zone; after the Zone group is added to the reusable list, the remaining space of the Zone can continue to be reused by other second files.

[0062] It should be noted that in order to avoid the loss of information in the reuse file list and the Zone group reuse list after the metadata management server powers off, when the reuse file list and the Zone group reuse list in the metadata management server change, the reuse file list and the Zone group reuse list can be saved to the log data, and the log data can be sent to the target metadata management server.

[0063] That is to say, the addition, deletion, and update of the reuse file list and the Zone group reuse list need to record real-time logs and synchronize them to other standby metadata management servers. Once the metadata management server restarts, the data in the reuse list can be restored according to the log data.

[0064] Obviously, the above-described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. To better understand the above file writing method, the following will describe the above process in conjunction with embodiments, but it is not used to limit the technical solutions of the embodiments of the present invention. Specifically:

[0065] In an alternative embodiment, in the field of video security, small files (equivalent to the first file in the above embodiment) mainly include alarm videos, alarm pictures, checkpoint pictures, and face capture pictures. The alarm videos record real-time video streams, and the file sizes cannot be determined during writing. Generally, their average size is 10 MB and the maximum is 100 MB; pictures may occupy a space of 4K. The Zone in a Shingled Magnetic Recording (SMR) disk has a fixed size of 256 MB. The SMR disk only supports sequential writing and does not support random writing. In the SMR environment, we stipulate that the large file after merging small files cannot cross Zones, that is, the maximum data block of the large file on the data node is only 256 MB. In the embodiment of the present application, the data node is responsible for the management of multiple SMR disks, and the metadata server is responsible for the allocation of data nodes and the aggregation of data information. When writing a large file, according to the EC type of the file client, its data is distributed among multiple data nodes to ensure the recoverability and integrity of the data when a data node or disk is damaged.

[0066] Figure 3 It is the flowchart (two) of the file writing method according to the embodiment of the present invention. The specific steps are as follows:

[0067] Step S302: The small file client applies for a reusable file (equivalent to the second file in the above embodiment);

[0068] Step S304: The metadata management server determines whether the reusable file list (equivalent to the reusable file list in the above embodiment) is empty. If it is, execute Step S310; if not, execute Step S306;

[0069] Step S306: Determine whether the reusable file in the reusable file list meets the reusable conditions. If it does, execute Step S308; if not, execute Step S310;

[0070] Step S308: Return the data node group and Zone group number of the reusable file;

[0071] Step S310: The metadata management server creates a new large file (equivalent to the second file in the above embodiment);

[0072] Step S312: Determine whether the reusable Zone group list (equivalent to the Zone group reuse list in the above embodiment) is empty. If it is, execute Step S320; if not, execute Step S314;

[0073] Step S314: Determine whether the Zone groups in the Zone group reusable list meet the reusable conditions. If so, execute Step S316; if not, execute Step S320;

[0074] Step S316: Retrieve the Zone groups from the reusable Zone group list;

[0075] Step S318: Return the data node group and Zone group number of the Zone group;

[0076] Step S320: Generate a new file number;

[0077] Step S322: The load balancing module of the metadata management server allocates a new data node group;

[0078] Step S324: The load balancing module of the metadata management server allocates a new Zone group number;

[0079] Step S326: Return the data node group and Zone group number of the new file.

[0080] To better understand the above steps, specifically: When the small file client writes small files such as alarm videos and alarm pictures, it first applies to the metadata management server for a reusable large file. The metadata management server obtains a reusable large file from the reusable file list, and the following conditions need to be met when obtaining: 1. The EC erasure code types of the small file and the reusable file are the same; 2. The small file and the reusable file are in the same file object bucket and have the same file attributes, such as life cycle and readable permissions. If there is a reusable large file, the metadata management server directly returns the data node group and Zone group number of the file to the small file client. If there is no reusable large file for the small file to write, the metadata management server generates a new large file for small file merging. The metadata management server needs to allocate a group of data nodes and Zone group numbers for the large file according to the EC erasure code of the first written small file. Among them, the Zone group number consists of a string of randomly generated unique identification numbers. Before the metadata management server allocates a new data node group for the large file, it needs to first obtain reusable Zone groups from the Zone group reusable list. The following conditions must also be met when obtaining: 1. The newly applied large file and the Zone group have the same EC erasure code type; 2. The file objects stored in the newly applied large file and the Zone group are in the same object bucket; 3. The Zone group is used for small file storage. It should be noted that if there is no reusable Zone group, the load balancing module of the metadata management server allocates a new group of data nodes and a new Zone group according to the load balancing of the data nodes.

[0081] Further, after the small file client obtains the data node group and the Zone group number, it divides the data into data blocks and writes them to the corresponding data nodes in an aligned manner. When writing, it needs to send the Zone group number to the data nodes. The data nodes query the local cache according to the Zone group number to confirm whether the Zone group number already exists. If it does not exist, they select a real disk according to the load balancing algorithm of the data nodes and write the data. After the data is successfully written, the information including the disk and the Zone group number information is refreshed to the metadata database of the data nodes for persistence. This database is usually RocksDB. If it is confirmed that the Zone group number exists, the data is appended to the Zone of the corresponding disk.

[0082] Figure 4 It is the flowchart (III) of the file writing method according to an embodiment of the present invention. The specific steps are as follows:

[0083] Step S402: Report metadata information;

[0084] Step S404: Report the information of the data block and the information of the Zone;

[0085] Step S406: Update the Zone to the Zone group management cache;

[0086] Step S408: Update the data block to the file object cache;

[0087] Step S410: The metadata management server determines whether all the file object data blocks have been reported. If so, execute Step S412;

[0088] Step S412: Determine whether the file can be reused. If so, execute Step S414. If not, execute Step S416;

[0089] Step S414: Add the second file to the reusable file list;

[0090] Step S416: The metadata management server determines whether all the Zone groups have been reported. If so, execute Step S418;

[0091] Step S418: Determine whether the Zone group corresponding to the second file can be reused. If so, execute Step S420. If not, execute Step S422;

[0092] Step S420: Add the Zone group corresponding to the second file to the Zone group reusable list;

[0093] Step S422: End;

[0094] For better understanding, specifically: when the small file client writes an alarm video, it will ensure that the size of the small file does not exceed 100 MB. When the file is about to reach 100 MB, a new file will be switched for writing; after a small file is written, the small file client needs to notify the data node and the metadata management server of the file closure in sequence. After the file is closed, the data node reports the metadata information of the data block and the metadata information of the Zone to the metadata management server. Among them, the metadata information of the Zone includes the Zone group number, the total length of the Zone, the used length of the Zone, and the creation time of the Zone. The metadata management server needs to wait until all the Zone information on all data nodes under the Zone group number is reported before it can determine that the Zone group writing is completed. At the same time, for large files, it is also necessary to wait until all the metadata information of all data blocks is reported before it can be considered that a single IO write of the large file is completed. After that, the metadata management server first judges whether the large file can be reused again according to the large file reuse rule. The judgment conditions for the large file to be reusable are: 1. The large file is within the reuse cycle. The reuse cycle of the large file is set to 1 day to avoid too long a time span between the earliest written small file and the latest written small file in the same large file. This time period is default configurable; 2. The length of each data block of the large file is equal to ensure that the written data blocks are all aligned; 3. The large file switches from the writing state to the normal closed state to avoid the large file being allocated for reuse during the writing process; 4. The remaining space of the SMR disk Zone corresponding to the large file is greater than 100 MB to ensure that at least one small file can be completely written. If the large file can continue to be reused, the large file will be added to the reusable file list. When the large file cannot continue to be reused, the Zone group where the large file is located will be judged for the reusability of the Zone group. The Zone group needs to meet the following conditions to be added to the reusable list: 1. The Zone group is written in alignment; 2. All data nodes in the Zone group are online and valid; 3. The Zone group is within the reusable cycle. The default reusable cycle time of the Zone group is set to 7 days; 4. There is still remaining writable space in the Zone; after the Zone group is added to the reusable list, the remaining space of the Zone can continue to be reused by other large files. To avoid the loss of the file reusable list and the Zone group reusable list information after the metadata management server power-off, the addition, deletion, and update of the file reusable list and the Zone group reusable list need to record real-time logs and synchronize them to other standby metadata management servers. Once the metadata server restarts, the data in the reuse list can be restored according to the log information.

[0095] Figure 5It is a diagram of the server module of the file writing method according to an embodiment of the present invention. Specifically, according to functional division, the servers involved in small file storage include small file clients, metadata management servers, and data nodes. Among them, the small file client is mainly responsible for the writing and reading of front-end video streams and pictures. The metadata management server includes a file management module, a Zone group management module, a data node load balancing module, and a data block receiving and processing module. The data node includes a data block management module, a metadata database management, a Zone management module, and a disk writing and reading module.

[0096] Specifically, the complete process of writing small files to cloud storage: When writing small files, first apply for reusable files. If they do not exist, allocate new large files and apply for reusable Zone groups for the large files. If the reusable Zone groups do not exist either, generate data nodes and Zone group numbers according to the new application process for large files. When writing small files to a data node, the Zone group number needs to be carried. The data node determines whether it is a reusable Zone according to the Zone group number. If not, allocate a new Zone for writing; if so, append the writing. When the small file writing is completed and the file is closed, the data node needs to report the information of the Zone and data blocks to the metadata management server. The metadata management server preferentially determines whether the file is reusable and whether the Zone group is reusable according to the information.

[0097] In addition, the above embodiment (Conventional Magnetic Recording, abbreviated as CMR) solves the problem of fragmentation of metadata data blocks and also solves the problem of frequent IO caused by reading and writing small files in cloud storage. And for block storage devices, the file space is divided according to data blocks. In the SMR environment, once the data blocks cannot be guaranteed to be fully written, the remaining space in the Zone will be wasted. The embodiment of the present application proposes an optimization of reusing Zone groups on the basis of small file merging, and proposes a file reuse strategy and a Zone group reuse strategy, which improves the space utilization rate of the Zone of the SMR disk.

[0098] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present invention.

[0099] In this embodiment, a file writing device is further provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0100] Figure 6 is a structural block diagram of a file writing device according to an embodiment of the present invention. The device includes:

[0101] A receiving module 62, configured to receive an application request sent by a file client, where the application request carries first attribute information corresponding to a first file;

[0102] An obtaining module 64, configured to, in response to the application request, determine a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and obtain file information of the second file, where the second file is used for storage multiplexing with the first file;

[0103] It should be noted that the file information includes: a data node group, and a Zone group number.

[0104] A writing module 66, configured to send the file information of the second file to the file client, and instruct the file client to write the first file to a disk corresponding to the second file according to the file information.

[0105] Through the above modules, an application request sent by a file client is received, where the application request carries first attribute information corresponding to a first file; in response to the application request, a second file corresponding to second attribute information that matches the first attribute information is determined in a metadata management server, and file information of the second file is obtained, where the second file is used for storage multiplexing with the first file; the file information of the second file is sent to the file client, and the file client is instructed to write the first file to a disk corresponding to the second file according to the file information. That is to say, the metadata management server can select a second file that allows multiplexing for the first file according to the attributes of the first file sent by the file client, and send the file information of the second file to the file client. Furthermore, the file client can write the first file to a disk corresponding to the second file according to the file information. By adopting the above technical solution, the problem that in the process of merging and storing small files by the traditional method, a suitable file cannot be selected for the small files according to the attributes of the small files, resulting in low disk space utilization, is solved. Furthermore, a suitable file is selected for the small files for merging and storing, improving the disk space utilization.

[0106] Optionally, the obtaining module 64 is further configured to obtain a list of reusable files saved in the metadata management server, where the list of reusable files includes: multiple files for reuse with the first file storage; determine whether there is a second file in the list of reusable files that meets a first preset condition, where the first preset condition is used to indicate that the first attribute information matches the second attribute information; in the case of existence, determine the file that meets the first preset condition in the list of reusable files as the second file; in the case of non-existence, create the second file in the metadata management server.

[0107] The first preset condition includes: the error correction codes in the first attribute information are the same as the error correction codes in the second attribute information; when the first file and the second file are in the same file object bucket, the file attributes in the first attribute information are the same as the file attributes in the second attribute information.

[0108] Optionally, the obtaining module 64 is further configured to obtain a Zone group reuse list saved in the metadata management server; determine whether there is a Zone group in the Zone group reuse list that meets a second preset condition, and obtain the data node group corresponding to the Zone group and the Zone group number of the Zone group; create the second file according to the data node group corresponding to the Zone group and the Zone group number of the Zone group.

[0109] Optionally, the obtaining module 64 is further configured to determine that there is no Zone group in the Zone group reuse list that meets the second preset condition; instruct the load balancing module of the metadata management server to determine a data node group and a Zone group number, and create the second file according to the data node group and the Zone group number determined by the load balancing module.

[0110] The second preset condition includes: the second file and the Zone group in the Zone group reuse list have the same erasure code type; the files stored in the second file and the Zone group in the Zone group reuse list are in the same object bucket; the Zone group in the Zone group reuse list is used to store the first file.

[0111] Optionally, the writing module 66 is further configured to instruct the file client to divide the first file into multiple data blocks, and send the multiple data blocks and the Zone group number in the file information to multiple data nodes corresponding to the data node group in the file information; control each of the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number.

[0112] Optionally, the writing module 66 is further configured to control the data node to send the metadata information of the corresponding data block and the metadata information of the Zone to the metadata management server, so that the metadata management server saves the metadata information of the data block and the metadata information of the Zone to the file object cache.

[0113] Optionally, the writing module 66 is further configured to add the second file to the reusable file list of the metadata management server when the second file meets the third preset condition; when the second file does not meet the third preset condition, determine whether the Zone group where the second file is located meets the fourth preset condition; when the Zone group meets the fourth preset condition, add the Zone group to the Zone group reusable list of the metadata management server.

[0114] The third preset condition includes: the second file is within the reuse cycle; the length of each data block of the second file is equal; the status of the second file is the normal closed state; the remaining space of the disk Zone corresponding to the second file is greater than the first preset threshold.

[0115] The fourth preset condition includes: the Zone group can be aligned for writing; the data nodes of the Zone group are in the active state; the Zone group is within the reusable cycle; the remaining space of the Zone group is greater than the second preset threshold.

[0116] Optionally, the writing module 66 is further configured to save the reusable file list and the Zone group reusable list to the log data when the reusable file list and the Zone group reusable list in the metadata management server change, and send the log data to the target metadata management server.

[0117] Optionally, in this embodiment, the above storage medium may be set to store a computer program for performing the following steps:

[0118] S1. Receive an application request sent by a file client, where the application request carries first attribute information corresponding to a first file;

[0119] S2. In response to the application request, determine a second file corresponding to second attribute information matching the first attribute information in the metadata management server, and obtain file information of the second file, where the second file is used for storage reuse with the first file;

[0120] S3. Send the file information of the second file to the file client, and instruct the file client to write the first file to the disk corresponding to the second file according to the file information.

[0121] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as USB flash drives, read-only memory (ROM for short), random access memory (RAM for short), mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0122] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein.

[0123] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0124] Optionally, in this embodiment, the above processor may be configured to execute the following steps through a computer program:

[0125] S1. Receive an application request sent by a file client, where the application request carries first attribute information corresponding to a first file;

[0126] S2. In response to the application request, determine a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and obtain file information of the second file, where the second file is used for storage multiplexing with the first file;

[0127] S3. Send the file information of the second file to the file client, and instruct the file client to write the first file to a disk corresponding to the second file according to the file information.

[0128] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device, where the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0129] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein.

[0130] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed over a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0131] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A file writing method, characterized in that, Including: Receiving an application request sent by a file client, where the application request carries first attribute information corresponding to a first file; In response to the application request, determining a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and obtaining file information of the second file, where the second file is used for storage multiplexing with the first file; Sending the file information of the second file to the file client, and instructing the file client to write the first file to a disk corresponding to the second file according to the file information; Wherein, both the first attribute information and the second attribute information include: erasure code and file attribute; determining the second file corresponding to the second attribute information that matches the first attribute information includes: Determining the second file according to a multiplexed file list and a first preset condition, where the first preset condition includes: the erasure code in the first attribute information is the same as the erasure code in the second attribute information; when the first file and the second file are in the same file object bucket, the file attribute in the first attribute information is the same as the file attribute in the second attribute information, where the file attribute at least includes: lifecycle, readable permission.

2. The document writing method according to claim 1, wherein Determining the second file according to a multiplexed file list and a first preset condition includes: Obtaining the multiplexed file list saved in the metadata management server, where the multiplexed file list includes: multiple files used for storage multiplexing with the first file; Determining whether there is a second file in the multiplexed file list that meets the first preset condition, where the first preset condition is used to indicate that the first attribute information matches the second attribute information; If it exists, determining the file that meets the first preset condition in the multiplexed file list as the second file; If it does not exist, creating the second file in the metadata management server.

3. The method according to claim 2, wherein Creating the second file in the metadata management server includes: Obtaining a Zone group multiplexing list saved in the metadata management server; Determining that there is a Zone group in the Zone group multiplexing list that meets a second preset condition, and obtaining a data node group corresponding to the Zone group and a Zone group number of the Zone group; Creating the second file according to the data node group corresponding to the Zone group and the Zone group number of the Zone group.

4. The method according to claim 3, wherein After obtaining the Zone group multiplexing list saved in the metadata management server, the method further includes: Determining that there is no Zone group in the Zone group multiplexing list that meets the second preset condition; Instructing a load balancing module of the metadata management server to determine a data node group and a Zone group number, and creating the second file according to the data node group and the Zone group number determined by the load balancing module.

5. The method according to claim 3, characterized in that The second preset condition includes: the second file and the Zone group in the Zone group reuse list have the same erasure code type; the files stored in the second file and the Zone group in the Zone group reuse list are located in the same object bucket; the Zone group in the Zone group reuse list is used to store the first file.

6. The method according to claim 1, wherein Instructing the file client to write the first file to the disk corresponding to the second file according to the file information includes: Instructing the file client to divide the first file into multiple data blocks, and sending the multiple data blocks and the Zone group number in the file information to multiple data nodes corresponding to the data node group in the file information; Controlling each of the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number.

7. The method according to claim 6, characterized in that, After controlling each of the multiple data nodes to write the corresponding data block to the disk corresponding to the Zone group number according to the Zone group number, the method further includes: Controlling the data node to send the metadata information of the corresponding data block and the metadata information of the Zone to the metadata management server, so that the metadata management server saves the metadata information of the data block and the metadata information of the Zone to the file object cache.

8. The method according to claim 1, characterized in that After instructing the file client to write the first file to the disk corresponding to the second file according to the file information, the method further includes: Adding the second file to the reuse file list of the metadata management server when the second file meets the third preset condition; Determining whether the Zone group where the second file is located meets the fourth preset condition when the second file does not meet the third preset condition; Adding the Zone group to the Zone group reuse list of the metadata management server when the Zone group meets the fourth preset condition.

9. The method according to claim 8, wherein The third preset condition includes: The second file is within the reuse cycle; each data block of the second file has the same length; the status of the second file is the normal closed state; the remaining space of the disk Zone corresponding to the second file is greater than the first preset threshold.

10. The method according to claim 8, wherein The fourth preset condition includes: the Zone group can be aligned for writing; the data nodes of the Zone group are in the active state; the Zone group is within the reusable cycle; the remaining space of the Zone group is greater than the second preset threshold.

11. The method according to claim 8, wherein The method further includes: When the reuse file list and the Zone group reuse list in the metadata management server change, saving the reuse file list and the Zone group reuse list to the log data, and sending the log data to the target metadata management server.

12. A file writing device, characterized in that, Including: A receiving module, configured to receive an application request sent by a file client, where the application request carries first attribute information corresponding to a first file; An acquisition module, configured to, in response to the application request, determine a second file corresponding to second attribute information that matches the first attribute information in a metadata management server, and acquire file information of the second file, where the second file is used for storage multiplexing with the first file; A writing module, configured to send the file information of the second file to the file client, and instruct the file client to write the first file to a disk corresponding to the second file according to the file information; wherein both the first attribute information and the second attribute information include: erasure code and file attributes; The acquisition module is further configured to determine the second file according to a multiplexed file list and a first preset condition, where the first preset condition includes: the erasure code in the first attribute information is the same as the erasure code in the second attribute information; when the first file and the second file are located in the same file object bucket, the file attributes in the first attribute information are the same as the file attributes in the second attribute information, where the file attributes at least include: life cycle, readable permission.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, where the program, when running, executes the method described in any one of claims 1 to 11 above.

14. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 11 through the computer program.

Images