File storage task processing method and device, storage medium and electronic equipment
By monitoring and analyzing the data of file storage tasks, determining the task type and dynamically adjusting the working mode of file storage clusters, the problem of low efficiency in processing file storage tasks in distributed storage systems is solved, and targeted optimization and efficiency improvement are achieved.
Patent Information
- Application Number
- CN202411045437.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The distributed storage system is inefficient when processing file storage tasks, and users cannot accurately judge the type of storage services, resulting in the inability to perform targeted configurations, affecting processing efficiency.
By monitoring the operation execution data of the target file storage task, determining the task type, and setting the working mode of the file storage cluster according to the task type, including dynamic adjustment of the proportion of nodes for sequential read and write and random read and write.
The efficiency of file storage cluster processing file storage tasks is improved, and the processing efficiency is improved through targeted optimization of the working mode to adapt to different types of storage business needs.
Smart Images

Figure CN118916329A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of storage technology, and in particular, to a method, device, storage medium and electronic device for processing a file storage task. Background Art
[0002] With the development of computer technology, enterprises need to store more and more data, so distributed storage systems have been developed. However, current distributed storage products, such as file storage clusters, are not optimized for storage scenarios when they leave the factory. They use the default general configuration when processing all businesses. Although users can manually configure the distributed storage system, most customers cannot accurately determine what type of storage business they need to process, and thus cannot configure the file storage cluster according to the specific business scenario, resulting in low efficiency of the file storage cluster in processing file storage tasks.
[0003] Therefore, in the related art, there is a problem of low efficiency of the file storage cluster in processing file storage tasks.
[0004] In view of the problem of low efficiency of file storage cluster in processing file storage tasks in related technologies, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for processing a file storage task, so as to at least solve the problem of low efficiency of a file storage cluster in processing file storage tasks in the related art.
[0006] According to an embodiment of the present application, a method for processing a file storage task is provided, comprising: monitoring the operation execution data of a target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster comprises a plurality of file storage nodes, and each storage node corresponds to a storage server; determining a task type of the target file storage task based on the first monitoring data, wherein the task type comprises a first file storage task, a second file storage task, and a third file storage task; and setting a working mode of the file storage cluster when processing the target file storage task based on the task type.
[0007] In an exemplary embodiment, before monitoring the operation execution data of the target file storage task, the method also includes: monitoring the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster, the number of input and output operations per second of the file storage cluster; when it is determined that the first ratio is greater than a first preset ratio, and the number of input and output operations per second is greater than the first preset number, enabling a file storage task classification function of the file storage cluster, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth upper limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.
[0008] In an exemplary embodiment, the operation execution data of the target file storage task is monitored to obtain first monitoring data, including: obtaining the total number of files created by the file storage cluster when processing the target file processing task within a preset period; dividing the files created by the file storage cluster into first files and second files according to the size of the files created by the file storage cluster, wherein the file size of the first file is larger than a preset storage capacity, and the file size of the second file is smaller than the preset storage capacity, and obtaining the number of first files and the number of second files in the total number of files; obtaining the number of times the file storage cluster writes to the first file within the preset period to obtain the number of first file write operations, and obtaining the number of times the file storage cluster writes to the second file to obtain the number of second file write operations; obtaining the number of times the file storage cluster writes to the second file within the preset period Obtain the number of times the file storage cluster performs read operations on the first file to obtain the number of first file read operations, and obtain the number of times the file storage cluster performs read operations on the second file to obtain the number of second file read operations; obtain the total number of first file operations and the total number of second file operations, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; determine the total number of files, the first file number, the second file number, the first file write operation number, the second file write operation number, the first file read operation number, the second file read operation number, the total number of first file operations, and the total number of second file operations as the first monitoring data.
[0009] In an exemplary embodiment, determining the task type of the target file storage task based on the first monitoring data includes: in a case where it is determined that the second ratio is greater than a second preset ratio, determining the target file storage task as the first file storage task, wherein the second ratio is the ratio of the first number of files to the total number of files, and the total number of files is equal to the sum of the first number of files and the second number of files; in a case where it is determined that the second ratio is less than a third preset ratio, determining the target file storage task as the second file storage task, wherein the third preset ratio is less than the second preset ratio.
[0010] In an exemplary embodiment, the method further includes: when it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset value, comparing the third ratio and the fourth preset ratio; when it is determined that the third ratio is greater than the fourth preset ratio, determining the target file storage task as the first file storage task, wherein the third ratio is the ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; when it is determined that the third ratio is less than a fifth preset ratio, determining the target file storage task as the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; when it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio, determining the target file storage task as the third file storage task.
[0011] In an exemplary embodiment, before setting the working mode of the file storage cluster when processing the target file storage task based on the task type, the method also includes: configuring multiple working modes for the file storage cluster, the multiple working modes including at least a first working mode, a second working mode, and a third working mode, wherein, in the first working mode, the file storage nodes with a sixth preset ratio in the file storage cluster are used for sequential reading and writing, in the second working mode, the file storage nodes with the sixth preset ratio in the file storage cluster are used for random reading and writing, and in the third working mode, the proportion of file storage nodes used for sequential reading and writing and the proportion of file storage nodes used for random reading and writing in the file storage cluster are dynamically adjusted.
[0012] In an exemplary embodiment, the working mode of the file storage cluster when processing the target file storage task is set based on the task type, including: when it is determined that the target file storage task is the first file storage task, setting the working mode of the file storage cluster to the first working mode; when it is determined that the target file storage task is the second file storage task, setting the working mode of the file storage cluster to the second working mode; when it is determined that the target file storage task is the third file storage task, setting the working mode of the file storage cluster to the third working mode.
[0013] According to another embodiment of the present application, a device for processing a file storage task is provided, including: a monitoring module, used to monitor the operation execution data of a target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, and the file storage cluster includes multiple file storage nodes, each storage node corresponds to a storage server; a determination module, used to determine the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task; a setting module, used to set the working mode of the file storage cluster when processing the target file storage task based on the task type.
[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.
[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiment are implemented.
[0017] Through the present application, the operation execution data of the target file storage task can be monitored, the task type of the target file storage task can be determined based on the monitoring data, and the working mode of the file storage cluster when processing the target file storage task can be set according to different task types. Therefore, the problem of low efficiency of the file storage cluster in processing file storage tasks in the related art can be solved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a hardware structure block diagram of a server device of a method for processing a file storage task in an embodiment of the present application;
[0019] Figure 2 is a flowchart of a method for processing a file storage task according to an embodiment of the present application;
[0020] Figure 3 is a schematic flow chart of a method for processing a file storage task according to an embodiment of the present application (I);
[0021] Figure 4 is a schematic flow chart of a method for processing a file storage task according to an embodiment of the present application (II);
[0022] Figure 5 It is a structural block diagram of a device for processing file storage tasks according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0025] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 1 is a hardware structure block diagram of a server device of a method for processing a file storage task according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0026] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for processing the file storage task in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the method for processing the above-mentioned file storage task. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0027] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0028] In this embodiment, a method for processing a file storage task is provided. Figure 2 is a flowchart of a method for processing a file storage task according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:
[0029] Step S202, monitoring the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes a plurality of file storage nodes, and each storage node corresponds to a storage server;
[0030] Optionally, in the above step S202, the operation execution data of the target file storage task includes operations such as creating a file, reading a file, and writing a file.
[0031] Step S204: determining a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;
[0032] Optionally, in the above step S204, for example, the first file storage task is a large file storage task, the second file storage task is a small file storage task, and the third file storage task is a mixed file storage task.
[0033] Step S206: Setting a working mode of the file storage cluster when processing the target file storage task based on the task type.
[0034] Through the above steps, the operation execution data of the target file storage task can be monitored, the task type of the target file storage task can be determined based on the monitoring data, and the working mode of the file storage cluster when processing the target file storage task can be set according to different task types. Therefore, the problem of low efficiency of the file storage cluster in processing file storage tasks in the related art can be solved.
[0035] In an exemplary embodiment, before monitoring the operation execution data of the target file storage task, the method also includes: monitoring the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster, the number of input and output operations per second of the file storage cluster; when it is determined that the first ratio is greater than a first preset ratio, and the number of input and output operations per second is greater than the first preset number, enabling a file storage task classification function of the file storage cluster, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth upper limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.
[0036] Optionally, in the above embodiment, the method also includes: when it is determined that the file storage cluster has turned on the file storage task classification function, monitoring the performance data of the file storage cluster to obtain third monitoring data, the third monitoring data including: the bandwidth of the file storage cluster, the number of input and output operations per second of the file storage cluster, and when it is determined that the first ratio is less than the first preset ratio and the number of input and output operations per second is less than the first preset number, turning off the file storage task classification function of the file storage cluster.
[0037] Optionally, in the above embodiment, for example, the bandwidth and IOPS (Input / Output Operations Per Second) of the file storage cluster are monitored. For example, the bandwidth upper limit is 300MB / s. When the IOPS is lower than 200 and the bandwidth is lower than 100MB / s, it means that the file storage cluster meets the business performance requirements and no performance optimization is required. Therefore, the file storage task classification function of the file storage cluster will not be enabled. When the IOPS is higher than 200 (equivalent to the first preset number) and the bandwidth is higher than 100MB / s, the bandwidth occupancy exceeds one third (equivalent to the first preset ratio), and the file storage task classification function of the file storage cluster is enabled.
[0038] In an exemplary embodiment, the operation execution data of the target file storage task is monitored to obtain first monitoring data, including: obtaining the total number of files created by the file storage cluster when processing the target file processing task within a preset period; dividing the files created by the file storage cluster into first files and second files according to the size of the files created by the file storage cluster, wherein the file size of the first file is larger than a preset storage capacity, and the file size of the second file is smaller than the preset storage capacity, and obtaining the number of first files and the number of second files in the total number of files; obtaining the number of times the file storage cluster writes to the first file within the preset period to obtain the number of first file write operations, and obtaining the number of times the file storage cluster writes to the second file to obtain the number of second file write operations; obtaining the number of times the file storage cluster writes to the second file within the preset period Obtain the number of times the file storage cluster performs read operations on the first file to obtain the number of first file read operations, and obtain the number of times the file storage cluster performs read operations on the second file to obtain the number of second file read operations; obtain the total number of first file operations and the total number of second file operations, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; determine the total number of files, the first file number, the second file number, the first file write operation number, the second file write operation number, the first file read operation number, the second file read operation number, the total number of first file operations, and the total number of second file operations as the first monitoring data.
[0039] Optionally, in the above embodiment, the operation execution data of the target file storage task may be counted by the following steps:
[0040] The first step is to count the number of files created by the file storage cluster (equivalent to the total number of files) in a seven-day period (equivalent to the preset period);
[0041] The second step is to count the file sizes created by the file storage cluster, and divide the files into large files (equivalent to the first files) and small files (equivalent to the second files) with 1MB (equivalent to the preset storage capacity) as the dividing line, and then calculate the number of large files (equivalent to the number of first files) and the number of small files (equivalent to the number of second files);
[0042] The third step is to count the number of operations of two file types: large files and small files. Among them, file creation, modification and deletion are write operations, and the number of large file write operations (equivalent to the number of write operations of the first file) and the number of small file write operations (equivalent to the number of write operations of the second file) are obtained. Among them, file reading is a read operation, and the number of large file read operations (equivalent to the number of read operations of the first file) and the number of small file read operations (equivalent to the number of read operations of the second file) are obtained.
[0043] Through the above embodiments, the storage services can be monitored and analyzed in detail, and then the storage services can be classified based on the monitoring data, and targeted optimization can be performed for different storage service processing processes.
[0044] In an exemplary embodiment, determining the task type of the target file storage task based on the first monitoring data includes: in a case where it is determined that the second ratio is greater than a second preset ratio, determining the target file storage task as the first file storage task, wherein the second ratio is the ratio of the first number of files to the total number of files, and the total number of files is equal to the sum of the first number of files and the second number of files; in a case where it is determined that the second ratio is less than a third preset ratio, determining the target file storage task as the second file storage task, wherein the third preset ratio is less than the second preset ratio.
[0045] In an exemplary embodiment, the method further includes: when it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset value, comparing the third ratio and the fourth preset ratio; when it is determined that the third ratio is greater than the fourth preset ratio, determining the target file storage task as the first file storage task, wherein the third ratio is the ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; when it is determined that the third ratio is less than a fifth preset ratio, determining the target file storage task as the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; when it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio, determining the target file storage task as the third file storage task.
[0046] Optionally, in the above embodiment, for example, when the file storage cluster processes the target file storage task, the number of large files accounts for more than 95% of the total number of files (equivalent to the second preset ratio), the target file storage task is determined to be a large file task (equivalent to the first file storage task), and the number of large files accounts for less than 5% of the total number of files (equivalent to the third preset ratio), the target file storage task is determined to be a small file task (equivalent to the second file storage task). If the number of large files accounts for the total number of files, the number of operations of large file types and the number of operations of small file types are further analyzed, for example, the number of large file operations (equivalent to the total number of first file operations) accounts for more than 90% of the total number of operations (equivalent to the total number of operations) (equivalent to the fourth preset ratio), the target file storage task is determined to be a large file task, the number of large file operations accounts for less than 10% of the total number of operations (equivalent to the fifth preset ratio), the target file storage task is determined to be a small file task, and if the number of large file operations accounts for the total number of operations, the target file storage task is determined to be a mixed file task (equivalent to the third file storage task).
[0047] Through the above embodiments, the storage service can be monitored to determine the storage service type, and then the working mode of the file storage cluster can be adjusted according to the service type to achieve targeted optimization and improve the processing efficiency of the storage service.
[0048] In an exemplary embodiment, before setting the working mode of the file storage cluster when processing the target file storage task based on the task type, the method also includes: configuring multiple working modes for the file storage cluster, the multiple working modes including at least a first working mode, a second working mode, and a third working mode, wherein, in the first working mode, the file storage nodes with a sixth preset ratio in the file storage cluster are used for sequential reading and writing, in the second working mode, the file storage nodes with the sixth preset ratio in the file storage cluster are used for random reading and writing, and in the third working mode, the proportion of file storage nodes used for sequential reading and writing and the proportion of file storage nodes used for random reading and writing in the file storage cluster are dynamically adjusted.
[0049] Optionally, in the above embodiments, for example, the first working mode, the second working mode, and the third working mode are respectively the large file working mode, the small file working mode, and the mixed file working mode. When the storage server uses sequential reading and writing when processing the reading and writing of large files, it is more efficient, and when using random reading and writing when processing the reading and writing of small files, it is more efficient. Therefore, targeted optimization can be performed. For example, there are a total of 10 file storage nodes in the file storage cluster, that is, 10 storage servers. When processing large file storage tasks, it can be set to 1 server using random reading and writing to process small files, and 9 servers using sequential reading and writing to process large files. When processing small file storage tasks, it can be set to 9 servers using random reading and writing to process small files, and 1 server using sequential reading and writing to process large files. When processing mixed file storage tasks, the server read and write processing method can be dynamically adjusted to achieve higher processing efficiency.
[0050] In an exemplary embodiment, the working mode of the file storage cluster when processing the target file storage task is set based on the task type, including: when it is determined that the target file storage task is the first file storage task, setting the working mode of the file storage cluster to the first working mode; when it is determined that the target file storage task is the second file storage task, setting the working mode of the file storage cluster to the second working mode; when it is determined that the target file storage task is the third file storage task, setting the working mode of the file storage cluster to the third working mode.
[0051] Obviously, the embodiments described above are only some embodiments of the present application, not all embodiments. In order to better understand the verification method of the above test case, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application, specifically:
[0052] In an alternative embodiment, Figure 31 is a schematic flow chart of a method for processing a file storage task according to an embodiment of the present application, which specifically includes the following steps:
[0053] S301: Turn on the business analysis function through switch settings;
[0054] Optionally, in the above step S301, the loads such as IOPS and bandwidth of the file storage cluster may be monitored. If the load data is low, it means that the file storage cluster meets the business performance requirements and does not need to be optimized for the business processing process; when the load data is high, it may cause a performance bottleneck in the file storage cluster, so the business analysis function is enabled to optimize the business processing process;
[0055] S302: Analyze the business type;
[0056] Optionally, in the above step S302, analyzing the business type requires collecting data such as the number of files involved in the business, file size, read and write operation records, and then dividing the business into different types based on the data characteristics, such as including but not limited to large file business type, small file business type, mixed file business type, etc.
[0057] S303: Output the business type analysis result.
[0058] Through the above steps, the business processing process can be optimized based on the business type analysis results to improve business processing efficiency.
[0059] In an alternative embodiment, Figure 4 2 is a schematic flow chart of a method for processing a file storage task according to an embodiment of the present application, which specifically includes the following steps:
[0060] S401: Monitor cluster performance, such as IOPS and bandwidth performance, through a monitoring module. If the performance meets the conditions, such as IOPS is higher than 200 and bandwidth is higher than 100 MB / s, start analyzing the target business (equivalent to the target file storage task);
[0061] S402: Counting the total number of files involved in the target business through a quantity statistics module;
[0062] S403: Counting the number of large files through a number counting module;
[0063] S404: Counting the number of file operations through an operation statistics module, where the operations specifically include file creation, reading, modification, and deletion operations;
[0064] S405: Based on the statistical data of step S403, the file size ratio is analyzed by a quantity analysis module, such as the ratio of large files and the ratio of small files;
[0065] S406: Based on the statistical data of step S404, the file operation ratio is analyzed by the operation analysis module, such as the operation ratio of large files and the operation ratio of small files;
[0066] Optionally, the above step S406 can be further refined, for example, obtaining the ratio of large file write operations to large file read operations in large file operations, and the ratio of small file write operations to small file operations in small file operations.
[0067] S407: Output the business type analysis results in a 7-day cycle.
[0068] Through the above embodiments, different services can be classified, and different working modes can be configured for the file storage cluster. Then, the working mode can be adjusted based on the service type to improve service processing efficiency.
[0069] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0070] In the present embodiment, a processing device for file storage tasks is also provided, and the system is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.
[0071] Figure 5 is a structural block diagram of a device for processing file storage tasks according to an embodiment of the present application, such as Figure 5 As shown, the device comprises:
[0072] A monitoring module 52 is used to monitor the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server;
[0073] A determination module 54, configured to determine a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;
[0074] The setting module 56 is used to set the working mode of the file storage cluster when processing the target file storage task based on the task type.
[0075] Through the above device, the operation execution data of the target file storage task can be monitored, the task type of the target file storage task can be determined based on the monitoring data, and the working mode of the file storage cluster when processing the target file storage task can be set according to different task types. Therefore, the problem of low efficiency of the file storage cluster in processing file storage tasks in the related art can be solved.
[0076] In an exemplary embodiment, the monitoring module 52 is also used to: monitor the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster, and the number of input and output operations per second of the file storage cluster; when it is determined that the first ratio is greater than a first preset ratio, and the number of input and output operations per second is greater than the first preset number, enable the file storage task classification function of the file storage cluster, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth upper limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.
[0077] In an exemplary embodiment, the monitoring module 52 is further used to: obtain the total number of files created by the file storage cluster when processing the target file processing task within a preset period; divide the files created by the file storage cluster into a first file and a second file according to the size of the files created by the file storage cluster, wherein the file size of the first file is larger than the preset storage capacity, and the file size of the second file is smaller than the preset storage capacity, and obtain the number of first files and the number of second files in the total number of files; obtain the number of times the file storage cluster performs write operations on the first file within the preset period to obtain the number of first file write operations, and obtain the number of times the file storage cluster performs write operations on the second file to obtain the number of second file write operations; obtain the number of times the file storage cluster performs write operations on the first file within the preset period The number of times the first file is read is obtained, and the number of times the file storage cluster performs read operations on the second file is obtained, and the number of second file read operations is obtained; the total number of first file operations and the total number of second file operations are obtained, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; the total number of files, the first file number, the second file number, the first file write operation number, the second file write operation number, the first file read operation number, the second file read operation number, the total number of first file operations, and the total number of second file operations are determined as the first monitoring data.
[0078] In an exemplary embodiment, the determination module 54 is further used to: determine the target file storage task as the first file storage task when it is determined that the second ratio is greater than a second preset ratio, wherein the second ratio is the ratio of the first number of files to the total number of files, and the total number of files is equal to the sum of the first number of files and the second number of files; and determine the target file storage task as the second file storage task when it is determined that the second ratio is less than a third preset ratio, wherein the third preset ratio is less than the second preset ratio.
[0079] In an exemplary embodiment, the determination module 54 is further used to: compare the third ratio with a fourth preset ratio when it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset ratio; determine the target file storage task as the first file storage task when it is determined that the third ratio is greater than the fourth preset ratio, wherein the third ratio is the ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; determine the target file storage task as the second file storage task when it is determined that the third ratio is less than a fifth preset ratio, wherein the fifth preset ratio is less than the fourth preset ratio; determine the target file storage task as the third file storage task when it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio.
[0080] In an exemplary embodiment, the setting module 56 is also used to: configure multiple working modes for the file storage cluster, the multiple working modes include at least a first working mode, a second working mode, and a third working mode, wherein, in the first working mode, the file storage nodes with a sixth preset ratio in the file storage cluster are used for sequential reading and writing, in the second working mode, the file storage nodes with the sixth preset ratio in the file storage cluster are used for random reading and writing, and in the third working mode, the proportion of file storage nodes used for sequential reading and writing and the proportion of file storage nodes used for random reading and writing in the file storage cluster are dynamically adjusted.
[0081] In an exemplary embodiment, the setting module 56 is also used to: when it is determined that the target file storage task is the first file storage task, set the working mode of the file storage cluster to the first working mode; when it is determined that the target file storage task is the second file storage task, set the working mode of the file storage cluster to the second working mode; when it is determined that the target file storage task is the third file storage task, set the working mode of the file storage cluster to the third working mode.
[0082] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.
[0083] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.
[0084] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:
[0085] S1, monitoring the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes a plurality of file storage nodes, and each storage node corresponds to a storage server;
[0086] S2, determining a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;
[0087] S3: Setting a working mode of the file storage cluster when processing the target file storage task based on the task type.
[0088] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0089] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0090] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:
[0091] S1, monitoring the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes a plurality of file storage nodes, and each storage node corresponds to a storage server;
[0092] S2, determining a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;
[0093] S3: Setting a working mode of the file storage cluster when processing the target file storage task based on the task type.
[0094] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.
[0095] Optionally, in this embodiment, the computer program may be configured to perform the following steps by means of the computer program:
[0096] S1, monitoring the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes a plurality of file storage nodes, and each storage node corresponds to a storage server;
[0097] S2, determining a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;
[0098] S3: Setting a working mode of the file storage cluster when processing the target file storage task based on the task type.
[0099] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.
[0100] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
[0101] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for processing a file storage task, characterized in that: include: Monitoring the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes a plurality of file storage nodes, and each storage node corresponds to a storage server; Determine a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task; The working mode of the file storage cluster when processing the target file storage task is set based on the task type.
2. The method according to claim 1, characterized in that Before monitoring the operation execution data of the target file storage task, the method further includes: Monitoring the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster and the number of input and output operations per second of the file storage cluster; When it is determined that the first ratio is greater than the first preset ratio and the number of input and output operations per second is greater than the first preset number, the file storage task classification function of the file storage cluster is enabled, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth upper limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.
3. The method according to claim 1, characterized in that The operation execution data of the target file storage task is monitored to obtain first monitoring data, including: Acquire the total number of files created by the file storage cluster when processing the target file processing task within a preset period; Dividing the files created by the file storage cluster into first files and second files according to the file sizes created by the file storage cluster, wherein the file size of the first file is larger than a preset storage capacity, and the file size of the second file is smaller than the preset storage capacity, and obtaining the number of first files and the number of second files in the total number of files; Acquire the number of write operations performed by the file storage cluster on the first file within the preset period to obtain the first file write operation number, and acquire the number of write operations performed by the file storage cluster on the second file to obtain the second file write operation number; Acquire the number of times the file storage cluster performs read operations on the first file within the preset period to obtain the first file read operation number, and acquire the number of times the file storage cluster performs read operations on the second file to obtain the second file read operation number; Obtaining a total number of first file operations and a total number of second file operations, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; The total number of files, the first number of files, the second number of files, the number of write operations on the first file, the number of write operations on the second file, the number of read operations on the first file, the number of read operations on the second file, the total number of operations on the first file, and the total number of operations on the second file are determined as the first monitoring data.
4. The method according to claim 3, characterized in that Determining the task type of the target file storage task based on the first monitoring data includes: In the case where it is determined that the second ratio is greater than a second preset ratio, determining the target file storage task as the first file storage task, wherein the second ratio is a ratio of the first file quantity to the total number of files, and the total number of files is equal to the sum of the first file quantity and the second file quantity; In the case where it is determined that the second ratio is smaller than a third preset ratio, the target file storage task is determined as the second file storage task, wherein the third preset ratio is smaller than the second preset ratio.
5. The method according to claim 4, characterized in that The method further comprises: When it is determined that the second ratio is smaller than the second preset ratio and the second ratio is larger than the third preset value, comparing the third ratio with a fourth preset ratio; In the case where it is determined that the third ratio is greater than the fourth preset ratio, determining the target file storage task as the first file storage task, wherein the third ratio is a ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; In the case where it is determined that the third ratio is less than a fifth preset ratio, determining the target file storage task as the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; When it is determined that the third ratio is smaller than the fourth preset ratio and the third ratio is larger than the fifth preset ratio, the target file storage task is determined to be the third file storage task.
6. The method according to claim 1, characterized in that Before setting the working mode of the file storage cluster when processing the target file storage task based on the task type, the method further includes: A plurality of working modes are configured for the file storage cluster, and the plurality of working modes include at least a first working mode, a second working mode, and a third working mode, wherein in the first working mode, the file storage nodes with a sixth preset ratio in the file storage cluster are used for sequential reading and writing, in the second working mode, the file storage nodes with the sixth preset ratio in the file storage cluster are used for random reading and writing, and in the third working mode, the proportion of the file storage nodes used for sequential reading and writing and the proportion of the file storage nodes used for random reading and writing in the file storage cluster are dynamically adjusted.
7. The method according to claim 6, characterized in that Setting a working mode of the file storage cluster when processing the target file storage task based on the task type includes: In a case where it is determined that the target file storage task is the first file storage task, setting the working mode of the file storage cluster to the first working mode; In a case where it is determined that the target file storage task is the second file storage task, setting the working mode of the file storage cluster to the second working mode; When it is determined that the target file storage task is the third file storage task, the working mode of the file storage cluster is set to the third working mode.
8. A device for processing file storage tasks, characterized in that: include: A monitoring module, configured to monitor the operation execution data of the target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes a plurality of file storage nodes, and each storage node corresponds to a storage server; A determination module, configured to determine a task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task; A setting module is used to set the working mode of the file storage cluster when processing the target file storage task based on the task type.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Testing method and device of small file reading and writing performance of distributed storage system
CN107480039A
Distributed storage file reading and writing method, device, platform and readable storage medium
CN110990339A
File storage method and device
CN117234435A
Data access method and device, distributed storage system and equipment and medium
CN117591038A
Data read-write method of cluster file system, cluster file system and equipment
CN117785821A
