Methods, apparatus, storage media, and electronic devices for file storage tasks

Abstract

This application provides a method, apparatus, storage medium, and electronic device for processing file storage tasks, relating to the field of storage technology. The method includes: monitoring the operation execution data of a target file storage task to obtain first monitoring data, wherein a file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, each storage node corresponding to a storage server; determining the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task; and setting the working mode of the file storage cluster when processing the target file storage task based on the task type. This solves the problem of low efficiency in processing file storage tasks by file storage clusters in related technologies.

Description

Technical Field

[0001] This application relates to the field of storage technology, and more specifically, to a method, apparatus, storage medium, and electronic device for processing file storage tasks. Background Technology

[0002] With the development of computer technology, enterprises need to store more and more data, thus leading to the development of distributed storage systems. However, current distributed storage products, such as file storage clusters, are not optimized for storage scenarios at the factory. They use default general configurations to handle all business operations. Although users can manually configure the distributed storage system, most customers cannot accurately determine the type of storage business they need to handle, and therefore cannot configure the file storage cluster according to the specific business scenario, resulting in low efficiency in processing file storage tasks.

[0003] Therefore, in related technologies, there is a problem of low efficiency in processing file storage tasks by file storage clusters.

[0004] There is still no effective solution to the problem of low efficiency in processing file storage tasks by file storage clusters in related technologies. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for processing file storage tasks, so as to at least solve the problem of low efficiency in processing file storage tasks by file storage clusters in related technologies.

[0006] According to one embodiment of this application, a method for processing a file storage task is provided, comprising: monitoring the operation execution data of a target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, each storage node corresponding to a storage server; determining the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task; and setting the working mode of the file storage cluster when processing the target file storage task based on the task type.

[0007] In an exemplary embodiment, before monitoring the operation execution data of the target file storage task, the method further includes: monitoring the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster and the number of input / output operations per second of the file storage cluster; and, if it is determined that a first ratio is greater than a first preset ratio and the number of input / output operations per second is greater than a first preset number, enabling the file storage task classification function of the file storage cluster, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.

[0008] In an exemplary embodiment, monitoring the operation execution data of the target file storage task to obtain first monitoring data includes: acquiring the total number of files created by the file storage cluster when processing the target file processing task within a preset period; dividing the files created by the file storage cluster into first files and second files according to the file size, wherein the file size of the first file is greater than a preset storage capacity and the file size of the second file is less than the preset storage capacity, and acquiring the number of first files and the number of second files in the total number of files; acquiring the number of times the file storage cluster performs write operations on the first file within the preset period, and acquiring the number of times the file storage cluster performs write operations on the second file, and acquiring the number of times the file storage cluster performs write operations on the second file, and acquiring the number of write operations on the second file within the preset period; acquiring the number of first file write operations ... The number of times the file storage cluster performs read operations on the first file is obtained to obtain the first file read operation count, and the number of times the file storage cluster performs read operations on the second file is obtained to obtain the second file read operation count; the total number of first file operations and the total number of second file operations are obtained, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; the total number of files, the number of first files, the number of second files, the number of first file write operations, the number of second file write operations, the number of first file read operations, the number of second file read operations, the total number of first file operations, and the total number of second file operations are determined as the first monitoring data.

[0009] In an exemplary embodiment, determining the task type of the target file storage task based on the first monitoring data includes: if a second ratio is determined to be greater than a second preset ratio, determining the target file storage task as the first file storage task, wherein the second ratio is the ratio of the number of the first files to the total number of files, and the total number of files is equal to the sum of the number of the first files and the number of the second files; if the second ratio is determined to be less than a third preset ratio, determining the target file storage task as the second file storage task, wherein the third preset ratio is less than the second preset ratio.

[0010] In one exemplary embodiment, the method further includes: when it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset value, comparing the third ratio and the fourth preset ratio; when it is determined that the third ratio is greater than the fourth preset ratio, determining the target file storage task as the first file storage task, wherein the third ratio is the ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; when it is determined that the third ratio is less than the fifth preset ratio, determining the target file storage task as the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; and when it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio, determining the target file storage task as the third file storage task.

[0011] In an exemplary embodiment, before setting the working mode of the file storage cluster when processing the target file storage task based on the task type, the method further includes: configuring multiple working modes for the file storage cluster, the multiple working modes including at least a first working mode, a second working mode, and a third working mode, wherein, in the first working mode, a sixth preset ratio of file storage nodes in the file storage cluster is used for sequential read and write; in the second working mode, the sixth preset ratio of file storage nodes in the file storage cluster is used for random read and write; and in the third working mode, the proportion of file storage nodes used for sequential read and write and the proportion of file storage nodes used for random read and write in the file storage cluster are dynamically adjusted.

[0012] In one exemplary embodiment, setting the working mode of the file storage cluster when processing the target file storage task based on the task type includes: when the target file storage task is determined to be the first file storage task, setting the working mode of the file storage cluster to the first working mode; when the target file storage task is determined to be the second file storage task, setting the working mode of the file storage cluster to the second working mode; and when the target file storage task is determined to be the third file storage task, setting the working mode of the file storage cluster to the third working mode.

[0013] According to another embodiment of this application, a file storage task processing apparatus is provided, comprising: a monitoring module, configured to monitor the operation execution data of a target file storage task to obtain first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, each storage node corresponding to a storage server; a determination module, configured to determine the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task; and a setting module, configured to set the working mode of the file storage cluster when processing the target file storage task based on the task type.

[0014] According to yet another embodiment of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and the computer program is configured to perform the steps in any of the above method embodiments when it is run.

[0015] According to yet another embodiment of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0016] According to yet another embodiment of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0017] This application allows for the monitoring of operation execution data for target file storage tasks, the determination of the task type based on the monitoring data, and the setting of the working mode of the file storage cluster when processing the target file storage task according to different task types. Therefore, it solves the problem of low efficiency in processing file storage tasks by file storage clusters in related technologies. Attached Figure Description

[0018] Figure 1 This is a hardware structure block diagram of a server device for a file storage task processing method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of a file storage task processing method according to an embodiment of this application;

[0020] Figure 3 This is a schematic flowchart (a) of a file storage task processing method according to an embodiment of this application;

[0021] Figure 4 This is a schematic flowchart (II) of a file storage task processing method according to an embodiment of this application;

[0022] Figure 5 This is a structural block diagram of a file storage task processing apparatus according to an embodiment of this application. Detailed Implementation

[0023] The embodiments of this application will be described in detail below with reference to the accompanying drawings and examples.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0025] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a file storage task processing method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0026] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the file storage task processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus realizing the above-mentioned file storage task processing method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0027] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0028] This embodiment provides a method for processing file storage tasks. Figure 2 This is a flowchart of a file storage task processing method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0029] Step S202: Monitor the operation execution data of the target file storage task to obtain the first monitoring data. The file storage cluster is used to process the file storage task. The file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server.

[0030] Optionally, in step S202 above, the operation execution data for the target file storage task includes operations such as creating a file, reading a file, and writing a file.

[0031] Step S204: Determine the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;

[0032] Optionally, in step S204 above, for example, the first file storage task is a large file storage task, the second file storage task is a small file storage task, and the third file storage task is a mixed file storage task.

[0033] Step S206: Set the working mode of the file storage cluster when processing the target file storage task based on the task type.

[0034] Through the above steps, the operation execution data of the target file storage task can be monitored. Based on the monitoring data, the task type of the target file storage task can be determined, and the working mode of the file storage cluster when processing the target file storage task can be set according to different task types. Therefore, the problem of low efficiency in processing file storage tasks by file storage clusters in related technologies can be solved.

[0035] In an exemplary embodiment, before monitoring the operation execution data of the target file storage task, the method further includes: monitoring the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster and the number of input / output operations per second of the file storage cluster; and, if it is determined that a first ratio is greater than a first preset ratio and the number of input / output operations per second is greater than a first preset number, enabling the file storage task classification function of the file storage cluster, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.

[0036] Optionally, in the above embodiments, the method further includes: when it is determined that the file storage cluster has enabled the file storage task classification function, monitoring the performance data of the file storage cluster to obtain third monitoring data, the third monitoring data including: the bandwidth of the file storage cluster, the number of input / output operations per second of the file storage cluster, and when it is determined that the first ratio is less than the first preset ratio and the number of input / output operations per second is less than the first preset number, disabling the file storage task classification function of the file storage cluster.

[0037] Optionally, in the above embodiments, for example, the bandwidth and IOPS (Input / Output Operations Per Second) of the file storage cluster are monitored. For example, if the bandwidth limit is 300MB / s, and the IOPS is below 200 and the bandwidth is below 100MB / s, it indicates that the file storage cluster meets the business performance requirements and no performance optimization is needed. Therefore, the file storage task classification function of the file storage cluster will not be enabled. If the IOPS is above 200 (equivalent to the first preset number) and the bandwidth is above 100MB / s, and the bandwidth usage exceeds one-third (equivalent to the first preset ratio), then the file storage task classification function of the file storage cluster will be enabled.

[0038] In an exemplary embodiment, monitoring the operation execution data of the target file storage task to obtain first monitoring data includes: acquiring the total number of files created by the file storage cluster when processing the target file processing task within a preset period; dividing the files created by the file storage cluster into first files and second files according to the file size, wherein the file size of the first file is greater than a preset storage capacity and the file size of the second file is less than the preset storage capacity, and acquiring the number of first files and the number of second files in the total number of files; acquiring the number of times the file storage cluster performs write operations on the first file within the preset period, and acquiring the number of times the file storage cluster performs write operations on the second file, and acquiring the number of times the file storage cluster performs write operations on the second file, and acquiring the number of write operations on the second file within the preset period; acquiring the number of first file write operations ... The number of times the file storage cluster performs read operations on the first file is obtained to obtain the first file read operation count, and the number of times the file storage cluster performs read operations on the second file is obtained to obtain the second file read operation count; the total number of first file operations and the total number of second file operations are obtained, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; the total number of files, the number of first files, the number of second files, the number of first file write operations, the number of second file write operations, the number of first file read operations, the number of second file read operations, the total number of first file operations, and the total number of second file operations are determined as the first monitoring data.

[0039] Optionally, in the above embodiments, the operation execution data of the target file storage task can be statistically analyzed through the following steps:

[0040] The first step is to count the number of files created by the file storage cluster, Sum (equivalent to the total number of files), over a seven-day period (equivalent to a preset period).

[0041] The second step is to count the file sizes created by the file storage cluster, divide the files into large files (equivalent to the first file) and small files (equivalent to the second file) with 1MB (equivalent to the preset storage capacity) as the dividing line, and then calculate the number of large files (equivalent to the number of first files) and the number of small files (equivalent to the number of second files);

[0042] The third step is to count the number of operations on two file types: large files and small files. The creation, modification, and deletion of files are considered write operations. The number of write operations on large files (equivalent to the number of write operations on the first file) and the number of write operations on small files (equivalent to the number of write operations on the second file) are obtained. The reading of files is considered read operations. The number of read operations on large files (equivalent to the number of read operations on the first file) and the number of read operations on small files (equivalent to the number of read operations on the second file) are obtained.

[0043] Through the above embodiments, storage services can be monitored and analyzed in detail, and then storage services can be classified based on monitoring data, and targeted optimizations can be made for different storage service processing procedures.

[0044] In an exemplary embodiment, determining the task type of the target file storage task based on the first monitoring data includes: if a second ratio is determined to be greater than a second preset ratio, determining the target file storage task as the first file storage task, wherein the second ratio is the ratio of the number of the first files to the total number of files, and the total number of files is equal to the sum of the number of the first files and the number of the second files; if the second ratio is determined to be less than a third preset ratio, determining the target file storage task as the second file storage task, wherein the third preset ratio is less than the second preset ratio.

[0045] In one exemplary embodiment, the method further includes: when it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset value, comparing the third ratio and the fourth preset ratio; when it is determined that the third ratio is greater than the fourth preset ratio, determining the target file storage task as the first file storage task, wherein the third ratio is the ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; when it is determined that the third ratio is less than the fifth preset ratio, determining the target file storage task as the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; and when it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio, determining the target file storage task as the third file storage task.

[0046] Optionally, in the above embodiments, for example, when the file storage cluster processes a target file storage task, if the number of large files accounts for more than 95% of the total number of files (equivalent to a second preset ratio), the target file storage task is determined to be a large file task (equivalent to a first file storage task); if the number of large files accounts for less than 5% of the total number of files (equivalent to a third preset ratio), the target file storage task is determined to be a small file task (equivalent to a second file storage task). If the number of large files accounts for between 5% and 95% of the total number of files, the number of operations on large file types and the number of operations on small file types are further analyzed. For example, if the number of large file operations (equivalent to the total number of first file operations) accounts for more than 90% of the total number of operations (equivalent to the total number of operations), the target file storage task is determined to be a large file task; if the number of large file operations accounts for less than 10% of the total number of operations (equivalent to a fifth preset ratio), the target file storage task is determined to be a small file task; if the number of large file operations accounts for between 10% and 90% of the total number of operations, the target file storage task is determined to be a mixed file task (equivalent to a third file storage task).

[0047] Through the above embodiments, the storage service can be monitored to determine the type of storage service, and then the working mode of the file storage cluster can be adjusted according to the type of service to achieve targeted optimization and improve the processing efficiency of the storage service.

[0048] In an exemplary embodiment, before setting the working mode of the file storage cluster when processing the target file storage task based on the task type, the method further includes: configuring multiple working modes for the file storage cluster, the multiple working modes including at least a first working mode, a second working mode, and a third working mode, wherein, in the first working mode, a sixth preset ratio of file storage nodes in the file storage cluster is used for sequential read and write; in the second working mode, the sixth preset ratio of file storage nodes in the file storage cluster is used for random read and write; and in the third working mode, the proportion of file storage nodes used for sequential read and write and the proportion of file storage nodes used for random read and write in the file storage cluster are dynamically adjusted.

[0049] Optionally, in the above embodiments, for example, the first working mode, the second working mode, and the third working mode are respectively the large file working mode, the small file working mode, and the mixed file working mode. When the storage server is processing large file read and write operations, sequential read and write is more efficient, and when processing small file read and write operations, random read and write is more efficient. Therefore, targeted optimization can be performed. For example, if there are 10 file storage nodes in the file storage cluster, that is, 10 storage servers, when processing large file storage tasks, it can be set up so that 1 server uses random read and write to process small files and 9 servers use sequential read and write to process large files. When processing small file storage tasks, it can be set up so that 9 servers use random read and write to process small files and 1 server uses sequential read and write to process large files. When processing mixed file storage tasks, the server read and write processing method can be dynamically adjusted to achieve higher processing efficiency.

[0050] In one exemplary embodiment, setting the working mode of the file storage cluster when processing the target file storage task based on the task type includes: when the target file storage task is determined to be the first file storage task, setting the working mode of the file storage cluster to the first working mode; when the target file storage task is determined to be the second file storage task, setting the working mode of the file storage cluster to the second working mode; and when the target file storage task is determined to be the third file storage task, setting the working mode of the file storage cluster to the third working mode.

[0051] Obviously, the embodiments described above are only some embodiments of this application, and not all embodiments. To better understand the verification method for the above test cases, the following description, in conjunction with embodiments, illustrates the process, but is not intended to limit the technical solutions of the embodiments of this application. Specifically:

[0052] In an optional embodiment, Figure 3This is a schematic flowchart (I) of a file storage task processing method according to an embodiment of this application, specifically including the following steps:

[0053] S301: Enable the business analysis function via a switch setting;

[0054] Optionally, in step S301 above, the load of the file storage cluster, such as IOPS and bandwidth, can be monitored. If the load data is low, it means that the file storage cluster meets the business performance requirements and no optimization is needed for the business processing. When the load data is high, it may cause a performance bottleneck in the file storage cluster. Therefore, the business analysis function is enabled to optimize the business processing.

[0055] S302: Analyze the business type;

[0056] Optionally, in step S302 above, analyzing the business type requires collecting data such as the number of files involved in the business, file size, and read / write operation records. Then, based on the data characteristics, the business is divided into different types, such as including but not limited to large file business type, small file business type, and mixed file business type.

[0057] S303: Output the business type analysis results.

[0058] By following the steps above, the business processing can be optimized in a targeted manner based on the business type analysis results, thereby improving business processing efficiency.

[0059] In an optional embodiment, Figure 4 This is a schematic flowchart (II) of a file storage task processing method according to an embodiment of this application, specifically including the following steps:

[0060] S401: Monitor cluster performance, such as IOPS and bandwidth performance, through the monitoring module. If the performance meets the conditions, such as IOPS higher than 200 and bandwidth higher than 100MB / s, start analyzing the target business (equivalent to the target file storage task).

[0061] S402: Use the quantity statistics module to count the total number of files involved in the target business;

[0062] S403: Count the number of large files using the quantity statistics module;

[0063] S404: The operation statistics module counts the number of file operations, including file creation, reading, modification, and deletion operations.

[0064] S405: Based on the statistical data from step S403, analyze the file size ratio through the quantity analysis module, such as the percentage of large files and the percentage of small files;

[0065] S406: Based on the statistical data from step S404, analyze the proportion of file operations through the operation analysis module, such as the proportion of operations on large files and the proportion of operations on small files;

[0066] Optionally, step S406 above can be further refined, for example, the ratio of large file write operations to large file read operations in the operation of acquiring large files, and the ratio of small file write operations to small file read operations in the operation of small files.

[0067] S407: Output business type analysis results in a 7-day cycle.

[0068] Through the above embodiments, different services can be classified, and different working modes can be configured for the file storage cluster. Then, the working mode can be adjusted based on the service type to improve service processing efficiency.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0070] This embodiment also provides a file storage task processing apparatus. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0071] Figure 5 This is a structural block diagram of a file storage task processing apparatus according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0072] Monitoring module 52 is used to monitor the operation execution data of the target file storage task and obtain the first monitoring data. The file storage cluster is used to process the file storage task. The file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server.

[0073] The determining module 54 is used to determine the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;

[0074] Setting module 56 is used to set the working mode of the file storage cluster when processing the target file storage task based on the task type.

[0075] The aforementioned device can monitor the operation execution data of target file storage tasks, determine the task type of the target file storage task based on the monitoring data, and set the working mode of the file storage cluster when processing the target file storage task according to different task types. Therefore, it can solve the problem of low efficiency in processing file storage tasks by file storage clusters in related technologies.

[0076] In an exemplary embodiment, the monitoring module 52 is further configured to: monitor the performance data of the file storage cluster to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster and the number of input / output operations per second of the file storage cluster; and, if it is determined that a first ratio is greater than a first preset ratio and the number of input / output operations per second is greater than a first preset number, enable the file storage task classification function of the file storage cluster, wherein the first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.

[0077] In an exemplary embodiment, the monitoring module 52 is further configured to: acquire the total number of files created by the file storage cluster when processing the target file processing task within a preset period; divide the files created by the file storage cluster into first files and second files according to the file size, wherein the file size of the first file is greater than a preset storage capacity and the file size of the second file is less than the preset storage capacity, and acquire the number of first files and the number of second files in the total number of files; acquire the number of times the file storage cluster performs write operations on the first file within the preset period, to obtain the number of first file write operations, and acquire the number of times the file storage cluster performs write operations on the second file, to obtain the number of second file write operations; acquire the number of times the file storage cluster performs write operations on the second file within the preset period, to obtain the number of write operations on the second file; and acquire the number of times the file storage cluster performs write operations on the target file processing task within the preset period. The number of read operations performed on the first file is obtained to get the first file read operation count, and the number of read operations performed on the second file by the file storage cluster is obtained to get the second file read operation count; the total number of first file operations and the total number of second file operations are obtained, wherein the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations; the total number of files, the number of first files, the number of second files, the number of first file write operations, the number of second file write operations, the number of first file read operations, the number of second file read operations, the total number of first file operations, and the total number of second file operations are determined as the first monitoring data.

[0078] In an exemplary embodiment, the determining module 54 is further configured to: determine the target file storage task as the first file storage task when the second ratio is determined to be greater than the second preset ratio, wherein the second ratio is the ratio of the number of the first files to the total number of files, and the total number of files is equal to the sum of the number of the first files and the number of the second files; and determine the target file storage task as the second file storage task when the second ratio is determined to be less than the third preset ratio, wherein the third preset ratio is less than the second preset ratio.

[0079] In an exemplary embodiment, the determining module 54 is further configured to: compare a third ratio and a fourth preset ratio when the second ratio is determined to be less than the second preset ratio and greater than the third preset value; determine the target file storage task as the first file storage task when the third ratio is determined to be greater than the fourth preset ratio, wherein the third ratio is the ratio of the total number of first file operations to the total number of operations, and the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; determine the target file storage task as the second file storage task when the third ratio is determined to be less than a fifth preset ratio, wherein the fifth preset ratio is less than the fourth preset ratio; and determine the target file storage task as the third file storage task when the third ratio is determined to be less than the fourth preset ratio and greater than the fifth preset ratio.

[0080] In an exemplary embodiment, the setting module 56 is further configured to: configure multiple working modes for the file storage cluster, the multiple working modes including at least a first working mode, a second working mode, and a third working mode, wherein, in the first working mode, a sixth preset ratio of file storage nodes in the file storage cluster is used for sequential read and write; in the second working mode, the sixth preset ratio of file storage nodes in the file storage cluster is used for random read and write; and in the third working mode, dynamically adjust the proportion of file storage nodes used for sequential read and write and the proportion of file storage nodes used for random read and write in the file storage cluster.

[0081] In an exemplary embodiment, the setting module 56 is further configured to: set the working mode of the file storage cluster to the first working mode when the target file storage task is determined to be the first file storage task; set the working mode of the file storage cluster to the second working mode when the target file storage task is determined to be the second file storage task; and set the working mode of the file storage cluster to the third working mode when the target file storage task is determined to be the third file storage task.

[0082] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0083] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when run.

[0084] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0085] S1, monitor the operation execution data of the target file storage task to obtain the first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server;

[0086] S2, based on the first monitoring data, determine the task type of the target file storage task, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;

[0087] S3, Based on the task type, set the working mode of the file storage cluster when processing the target file storage task.

[0088] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0089] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0090] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0091] S1, monitor the operation execution data of the target file storage task to obtain the first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server;

[0092] S2, based on the first monitoring data, determine the task type of the target file storage task, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;

[0093] S3, Based on the task type, set the working mode of the file storage cluster when processing the target file storage task.

[0094] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0095] Optionally, in this embodiment, the computer program described above can be configured to perform the following steps:

[0096] S1, monitor the operation execution data of the target file storage task to obtain the first monitoring data, wherein the file storage cluster is used to process the file storage task, the file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server;

[0097] S2, based on the first monitoring data, determine the task type of the target file storage task, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task;

[0098] S3, Based on the task type, set the working mode of the file storage cluster when processing the target file storage task.

[0099] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0100] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be integrated on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0101] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method of processing a file storage task, characterized by, include: The operation execution data of the target file storage task is monitored to obtain the first monitoring data. The file storage cluster is used to process the file storage task. The file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server. Based on the first monitoring data, the task type of the target file storage task is determined, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task, the first file storage task is a large file storage task, the second file storage task is a small file storage task, and the third file storage task is a mixed file storage task; The working mode of the file storage cluster when processing the target file storage task is set based on the task type; The task type for determining the target file storage task based on the first monitoring data includes: If it is determined that the second ratio is greater than the second preset ratio, the target file storage task is determined as the first file storage task, wherein the second ratio is the ratio of the number of first files to the total number of files, and the total number of files is equal to the sum of the number of first files and the number of second files; If it is determined that the second ratio is less than the third preset ratio, the target file storage task is determined as the second file storage task, wherein the third preset ratio is less than the second preset ratio; If it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset ratio, then the third ratio and the fourth preset ratio are compared. If the third ratio is determined to be greater than the fourth preset ratio, the target file storage task is determined as the first file storage task. The third ratio is the ratio of the total number of first file operations to the total number of operations, where the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations. If the third ratio is determined to be less than the fifth preset ratio, the target file storage task is determined to be the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; If it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio, the target file storage task is determined as the third file storage task. Before setting the working mode of the file storage cluster when processing the target file storage task based on the task type, the method further includes: The file storage cluster is configured with multiple working modes, including at least a first working mode, a second working mode, and a third working mode. In the first working mode, a sixth preset ratio of file storage nodes in the file storage cluster is used for sequential read and write. In the second working mode, the sixth preset ratio of file storage nodes in the file storage cluster is used for random read and write. In the third working mode, the proportion of file storage nodes used for sequential read and write and the proportion of file storage nodes used for random read and write in the file storage cluster are dynamically adjusted. The method of setting the working mode of the file storage cluster when processing the target file storage task based on the task type includes: If the target file storage task is determined to be the first file storage task, the working mode of the file storage cluster is set to the first working mode; If the target file storage task is determined to be the second file storage task, the working mode of the file storage cluster is set to the second working mode; If the target file storage task is determined to be the third file storage task, the working mode of the file storage cluster is set to the third working mode.

2. The method of claim 1, wherein, Before monitoring the operation execution data of the target file storage task, the method further includes: The performance data of the file storage cluster is monitored to obtain second monitoring data, wherein the second monitoring data includes: the bandwidth of the file storage cluster and the number of input / output operations per second of the file storage cluster; When it is determined that the first ratio is greater than the first preset ratio and the number of input / output operations per second is greater than the first preset number, the file storage task classification function of the file storage cluster is enabled. The first ratio is the ratio of the bandwidth load of the file storage cluster to the bandwidth limit of the file storage cluster, and the file storage task classification function is used to determine the task type of the file storage task.

3. The method of claim 1, wherein, The execution data of the target file storage task is monitored to obtain the first monitoring data, including: Within a preset period, obtain the total number of files created by the file storage cluster when processing the target file processing task; Based on the file size created by the file storage cluster, the files created by the file storage cluster are divided into a first file and a second file, wherein the file size of the first file is greater than the preset storage capacity, and the file size of the second file is less than the preset storage capacity, and the number of the first file and the number of the second file are obtained from the total number of files; Within the preset period, the number of times the file storage cluster performs write operations on the first file is obtained to obtain the number of write operations on the first file, and the number of times the file storage cluster performs write operations on the second file is obtained to obtain the number of write operations on the second file. Within the preset period, the number of times the file storage cluster performs read operations on the first file is obtained to obtain the number of read operations on the first file, and the number of times the file storage cluster performs read operations on the second file is obtained to obtain the number of read operations on the second file. Get the total number of operations on the first file and the total number of operations on the second file; The total number of files, the number of the first file, the number of the second file, the number of write operations on the first file, the number of write operations on the second file, the number of read operations on the first file, the number of read operations on the second file, the total number of operations on the first file, and the total number of operations on the second file are determined as the first monitoring data.

4. An apparatus for processing a file storage task, characterized by comprising: include: The monitoring module is used to monitor the operation execution data of the target file storage task and obtain the first monitoring data. The file storage cluster is used to process the file storage task. The file storage cluster includes multiple file storage nodes, and each storage node corresponds to a storage server. The determination module is used to determine the task type of the target file storage task based on the first monitoring data, wherein the task type includes a first file storage task, a second file storage task, and a third file storage task, wherein the first file storage task is a large file storage task, the second file storage task is a small file storage task, and the third file storage task is a mixed file storage task; The configuration module is used to set the working mode of the file storage cluster when processing the target file storage task based on the task type. The determining module is used to determine the target file storage task as the first file storage task when the second ratio is greater than the second preset ratio, wherein the second ratio is the ratio of the number of first files to the total number of files, and the total number of files is equal to the sum of the number of first files and the number of second files; If it is determined that the second ratio is less than the third preset ratio, the target file storage task is determined as the second file storage task, wherein the third preset ratio is less than the second preset ratio; If it is determined that the second ratio is less than the second preset ratio and the second ratio is greater than the third preset ratio, then the third ratio and the fourth preset ratio are compared. If the third ratio is determined to be greater than the fourth preset ratio, the target file storage task is determined as the first file storage task. The third ratio is the ratio of the total number of first file operations to the total number of operations, where the total number of operations is equal to the sum of the total number of first file operations and the total number of second file operations; the total number of first file operations is equal to the sum of the number of first file write operations and the number of first file read operations, and the total number of second file operations is equal to the sum of the number of second file write operations and the number of second file read operations. If the third ratio is determined to be less than the fifth preset ratio, the target file storage task is determined to be the second file storage task, wherein the fifth preset ratio is less than the fourth preset ratio; If it is determined that the third ratio is less than the fourth preset ratio and the third ratio is greater than the fifth preset ratio, the target file storage task is determined as the third file storage task. The setting module is further configured to configure multiple working modes for the file storage cluster. The multiple working modes include at least a first working mode, a second working mode, and a third working mode. In the first working mode, a sixth preset ratio of file storage nodes in the file storage cluster is used for sequential read and write. In the second working mode, the sixth preset ratio of file storage nodes in the file storage cluster is used for random read and write. In the third working mode, the proportion of file storage nodes used for sequential read and write and the proportion of file storage nodes used for random read and write in the file storage cluster are dynamically adjusted. The setting module is further configured to set the working mode of the file storage cluster to the first working mode when the target file storage task is determined to be the first file storage task. If the target file storage task is determined to be the second file storage task, the working mode of the file storage cluster is set to the second working mode; If the target file storage task is determined to be the third file storage task, the working mode of the file storage cluster is set to the third working mode.

5. A computer readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of claims 1 to 3.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 3.

Images