Operation static sensing stripe optimization method and device, storage equipment and medium

Abstract

The invention discloses a job static perception stripe optimization method and device, a storage device and a medium, and relates to the technical field of computer optimizing.The method comprises the steps that offline data are analyzed, jobs related to write operation are screened out, the write I / O data amount of each job on each OST is counted and subjected to descending sort, the total write I / O data amount of each job is obtained through summation, and the total write I / O data amount of each job is calculated; attributes are added to the jobs, the jobs are marked as to-be-striped jobs, striping setting is carried out according to the applications, the number of the jobs of the applications is judged, and if the number of the jobs of the applications is not smaller than a fourth threshold value, subdirectory striping setting is carried out. Aiming at the problem of extremely unbalanced instantaneous load in a super computer optimization job OST, a stripe technology is applied to a storage system, a job capable of being subjected to stripe optimization is determined by exploring and analyzing a job I / O mode, in addition, static sensing file striping is provided, the strippable job can be quickly and effectively judged, and the efficiency of the strippable job is improved. Therefore, job distribution conditions of different users can be adapted.

Description

Technical Field

[0001] The present application relates to the field of computer optimization technology, and in particular to a method, apparatus, storage device and medium for job static-aware strip optimization. Background Art

[0002] As supercomputers continue to grow in size and complexity, their computing speeds have reached billions or even trillions of computing power per second. However, I / O performance has gradually become a major bottleneck. Traditional HPC applications and machine learning applications need to process large amounts of data, resulting in a large number of I / O requests and prolonged execution time. At the same time, the shift from compute-intensive jobs to data-intensive jobs has exacerbated the gap between compute and I / O performance. Although existing distributed computing technologies can reduce computing time, there is still a significant difference between compute and I / O performance, which has a negative impact on the overall performance and efficiency of supercomputers.

[0003] In order to address the negative impact, the process of optimizing I / O workloads has been widely developed, and many studies have used various techniques to obtain I / O workload data (I / O trace) from supercomputers to fully understand the application and system I / O behavior. In addition, some studies focus on analyzing the I / O behavior of workloads in specific fields, such as in the fields of machine learning or HPC file system environments, to explore the overall I / O trends and characteristics of the system, including I / O bandwidth, I / O requests, job arrival time, idle time, and I / O variability. In addition to analyzing application-level I / O logs, there are also storage server logs that are checked to evaluate application I / O characteristics. Some studies have found that the load on OST is highly unbalanced and a more aggressive file striping strategy can be used.

[0004] Striping technology such as Figure 1a and Figure 1b As shown, Figure 1a FIG. 1 shows a schematic diagram of a file before striping in the prior art. Figure 1b The schematic diagram of the file striping of the prior art is shown. File striping can disperse file data to multiple OSTs for writing. When writing a file, a file will be divided into multiple data stripes, and each data stripe will be written to a different OST. The use of file striping technology in supercomputers can effectively call underutilized OSTs and improve the performance and efficiency of the entire system. Although file striping can improve the performance and scalability of storage systems, in actual applications, many supercomputers do not use file striping by default. This is because file striping requires more management and configuration, as well as how to ensure data security and stability, making file striping difficult to use.

[0005] In addition, maintaining load balance of Object Storage Targets (OSTs) can effectively improve system I / O performance. However, OST load imbalance is an unavoidable challenge. Under the current traditional non-striping settings, multiple factors such as file distribution strategy, I / O access mode and file size will cause I / O requests to be concentrated on a few OSTs, making these OSTs overloaded while other OSTs are not fully utilized. This load imbalance will affect the efficiency and throughput of the system, and ultimately affect the overall performance of the supercomputer. Summary of the invention

[0006] The purpose of this application is to overcome the existing technical defects and provide a job static-aware striping optimization method, device, storage device and medium. By exploring and analyzing the job I / O mode, the jobs that can be striped are determined, and the jobs that can be striped can be quickly and effectively determined.

[0007] The purpose of this application is achieved through the following technical solutions: In a first aspect, the present application proposes a method for optimizing static-aware stripes of a job, the method comprising: Step S1: Analyze offline data and filter out jobs involving write operations through the write_bytes attribute; Step S2: Count the amount of write I / O data of each job on each OST and arrange them in descending order, and sum them to obtain the total amount of write I / O data of each job; Step S3: judge each job. If 80% of the total amount of write I / O data is concentrated in 20% of the write I / O data in descending order on each OST, add an attribute to the job and mark it as a job to be striped. Step S4, executing step S5 when the ratio of the jobs to be striped to the jobs involving write operations is not less than a first threshold; Step S5, determining whether the number of jobs is less than a second threshold, if so, executing step S6, if not, subdividing the jobs, performing striping according to the application, and then executing step S7; Step S6, striping the jobs according to the user home directory, and then executing step S9; Step S7, counting the number of jobs under each application and calculating the proportion of jobs that can be striped, and executing step S8; Step S8, judging the application, if the proportion of the stripeable jobs is not less than the third threshold, executing step S9; Step S9: judging the number of applied jobs, and performing sub-directory striping setting if the number of applied jobs is not less than a fourth threshold.

[0008] In one possible implementation, the subdirectory striping is set to set the stripe size to 1 MB and the stripe count to 4.

[0009] In a possible implementation manner, after step S9, the method further includes: Step S10: After the job is completed, the stripe setting of the main directory will be restored to the default setting.

[0010] In a possible implementation manner, the step S4 further includes: When the ratio of the jobs to be striped to the jobs involving write operations is less than the first threshold, no striping process is performed.

[0011] In a possible implementation, the application is each subdirectory in the WorkDir attribute of the job path of the data.

[0012] In a possible implementation, step S8 further includes: If the proportion of the stripeable jobs is less than the third threshold, the jobs starting with the name of the application in the job path WorkDir are not striped.

[0013] In a second aspect, the present application proposes a storage system operation static-aware stripe optimization device, the device comprising: The filtering module is used to analyze offline data and filter out jobs involving write operations through the write_bytes attribute; The summation module is used to count the amount of write I / O data of each job on each OST and sort them in descending order, and sum them up to get the total amount of write I / O data of each job; The marking module is used to judge each job. If 80% of the total write I / O data is concentrated in 20% of the write I / O data in descending order on each OST, an attribute is added to the job and marked as a job to be striped; A first judgment module, configured to execute a second judgment module when the ratio of jobs to be striped to jobs involving write operations is not less than a first threshold; A second judgment module is used to judge whether the number of jobs is less than a second threshold, and if so, execute the striping module; if not, subdivide the jobs, perform striping settings according to applications, and then execute the calculation module; The striping module is used to stripe the jobs according to the user's home directory, and then execute the striping setting module; A calculation module, used for counting the number of jobs under each application and calculating the proportion of jobs that can be striped, and executing a third judgment module; A third judgment module is used to judge the application, and if the proportion of the stripeable jobs is not less than a third threshold, execute the stripe setting module; The striping setting module is used to determine the number of jobs of the application, and perform sub-directory striping setting if the number of jobs of the application is not less than a fourth threshold.

[0014] In a possible implementation, the device further includes a recovery module, which is used to restore the stripe setting of the main directory to the default setting after the job is completed.

[0015] In a third aspect, the present application further proposes a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the job static-aware strip optimization method as described in any one of the first aspects.

[0016] In a fourth aspect, the present application further proposes a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the job static-aware strip optimization method as described in any one of the first aspects.

[0017] The above main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are many combinations based on the existing technology and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0018] The present application discloses a method, apparatus, storage device and medium for statically aware striping optimization of jobs. First, offline data is analyzed to screen out jobs involving write operations, the amount of write I / O data of each job on each OST is counted and arranged in descending order, and the total amount of write I / O data of each job is summed up. Attributes are added to the job, which is marked as a job to be striped, and striping is set according to the application. The number of jobs of the application is judged, and if the number of jobs of the application is not less than a fourth threshold, sub-directory striping is set. In order to solve the problem of extremely unbalanced instantaneous load in the OST of supercomputer optimization jobs, the present application applies striping technology to the storage system, and determines the jobs that can be striped by exploring and analyzing the job I / O mode. In addition, statically aware file striping is proposed, which can quickly and effectively determine the jobs that can be striped to adapt to the job distribution of different users. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1a A schematic diagram of the prior art file before striping is shown.

[0021] Figure 1b A schematic diagram of file striping in the prior art is shown.

[0022] Figure 2 A flow chart of a method for statically perceiving job strip optimization proposed in an embodiment of the present application is shown.

[0023] Figure 3 A job static-aware strip optimization framework proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0026] In the prior art, maintaining load balance of object storage targets can effectively improve system I / O performance. However, uneven load on OSTs is an unavoidable challenge. Under the current traditional non-striping settings, multiple factors such as file distribution strategy, I / O access mode, and file size will cause I / O requests to be concentrated on a few OSTs, making these OSTs overloaded while other OSTs are not fully utilized. This load imbalance will affect the efficiency and throughput of the system, and ultimately affect the overall performance of the supercomputer.

[0027] Therefore, in order to solve the problem of extremely unbalanced instantaneous load in the above-mentioned supercomputer optimization job OST, the embodiment of the present application proposes a method, device, storage device and medium for job static perception strip optimization. This method achieves effective improvement of I / O performance through intelligent analysis and optimization, while also ensuring data security and overall robustness of the system. This technology is particularly suitable for application scenarios that process large-scale data and high concurrent access, and is mainly divided into the following points: First, job analysis: We will first conduct an in-depth analysis of the running jobs to identify those tasks that can benefit from striping (striping is a data distribution strategy that enables concurrent reading and writing by storing data on multiple disks, thereby accelerating I / O operations).

[0028] Second, filter out jobs that can be optimized: After analysis, the jobs that are suitable for stripe optimization are selected. These jobs usually involve large amounts of data reading and writing and have high requirements for I / O performance.

[0029] Third, grouping strategy: jobs are further grouped according to user ID and file paths accessed by the jobs. Such grouping helps to fine-tune the implementation of striping, ensuring that jobs with similar relevance and access patterns are managed uniformly, thereby maximizing performance improvements and reducing potential conflicts.

[0030] Fourth, striping implementation: Striping configuration is performed for qualified users and jobs on the path. Data that may have been stored on a single disk is distributed on multiple disks, with each disk storing part of the data. In this way, when reading or writing data, it can be read or written from multiple disks at the same time, significantly improving I / O throughput.

[0031] Fifth, stripe recovery mechanism: In order to deal with possible stripe setting errors or system anomalies, a stripe recovery mechanism is also introduced. This is a set of safeguards that can automatically or manually restore to the previous stable state when an error occurs, ensuring data integrity and system stability, and reducing the risks caused by improper striping.

[0032] Please refer to Figure 2 , Figure 2 A schematic diagram of a process of a static-aware strip optimization method for a job proposed in an embodiment of the present application is shown, and the method includes: Step S1: Analyze offline data and filter out jobs involving write operations through the write_bytes attribute.

[0033] Load the offline data set into the analysis environment for analysis. The offline data may be stored in a database, data warehouse, or file system. Use query languages ​​(such as SQL, Pandas DataFrame operations, or other data analysis tools) to filter out all records whose write_bytes is non-zero or exceeds a certain threshold. Since write operations account for more than 90% and read operations account for a small proportion, the "write_bytes" attribute in the data is used to filter out jobs involving write operations.

[0034] In-depth analysis of the filtered data may include but is not limited to calculating the total write volume, average write size, time distribution of write operations, etc., which helps identify write operation patterns, hot spots, and potential performance bottlenecks. Based on the analysis results, it can provide a basis for further system optimization, resource allocation, or policy adjustment. For example, if it is found that write operations are extremely concentrated in certain time periods, it may be necessary to consider enhancing the storage system's write capacity during this period.

[0035] Step S2: Count the amount of write I / O data of each job on each OST and arrange them in descending order, and sum them up to obtain the total amount of write I / O data of each job.

[0036] Get the I / O operation logs or data records for all jobs on different OSTs. The data records can contain the job ID, OST identifier, and the number of read / write bytes for each I / O operation. For each record, identify the write operation (usually determined by the operation type field). You can group the data by job ID or OST and calculate the total write I / O data volume for each job on each OST. After summing up the write I / O data volume of each job on each OST, sort them in descending order by the total write I / O volume of the job. Sum the write I / O data volume of each job on all OSTs to get the total write I / O data volume for each job.

[0037] Step S3: judge each job. If 80% of the total write I / O data is concentrated in 20% of the write I / O data arranged in descending order on each OST, add attributes to the job and mark it as a job to be striped.

[0038] Determine the 20% of the total number of OSTs involved in each job, which carry a high write I / O traffic. For each job, calculate the sum of the write I / O data volume of the top 20% of OSTs after sorting. Compare the total write I / O volume of the top 20% of OSTs for each job with 80% of the total write I / O volume of the job. If the two are close or the former reaches the latter, it indicates that the write I / O traffic of the job is highly concentrated on a few OSTs, which is not conducive to load balancing. For jobs that meet the conditions (i.e., the write I / O traffic is highly concentrated), mark them as requiring stripe optimization so that improvement measures can be implemented later to balance the load.

[0039] After determining that the write I / O traffic of the job is concentrated on a few OSTs, these jobs are judged to need striping optimization to improve load balancing. To achieve this, a new attribute "type" is added to the data record of the job, and its value is marked as "true" or "false" depending on whether striping is required. The specific steps are as follows: Check each job analyzed previously one by one, and for each job, review the previously calculated concentration determination results. If a job is determined to have highly concentrated write I / O traffic (that is, 80% of the write I / O occurs on 20% of the OSTs), the "type" attribute of the job is assigned a value of "true", indicating that striping is required. Conversely, if the write I / O distribution is relatively uniform, the "type" attribute is set to "false", indicating that striping is not required.

[0040] Add or update the "type" attribute in the record of each job to ensure that subsequent system components or decision-making processes can identify and perform corresponding optimization measures based on the label.

[0041] Step S4: When the ratio of the jobs to be striped to the jobs involving write operations is not less than a first threshold, execute step S5.

[0042] Step S4 also includes: When the ratio of the jobs to be striped to the jobs involving write operations is less than the first threshold, no striping process is performed.

[0043] In the embodiment of the present application, the first threshold can be set to 95%, and all job records are grouped by user ID ("UID"), so that the data of each user can be analyzed separately. In each user group, the number of jobs with the "type" attribute as "true" is counted, and then divided by the total number of jobs under the user to obtain the ratio of jobs to be striped to jobs involving write operations (the ratio of jobs that can be striped optimized).

[0044] Compare whether the proportion of jobs that can be striped for each user reaches or exceeds 95%. If the proportion reaches or exceeds this threshold, it is considered that striping the jobs under this user will bring a significant performance improvement. Based on the above judgment, set a label or attribute for each user to indicate whether striping is performed on their jobs. If the proportion is ≥95%, it is marked as suitable for striping (for example, a "need_striping" attribute can be added at the user level and set to "true"); if the proportion is <95%, no striping is performed and it is marked as "need_striping=false".

[0045] Step S5, determine whether the number of jobs is less than the second threshold, if it is less than the second threshold, execute step S6, if it is not less than the second threshold, subdivide the jobs, set stripes according to the application, and then execute step S7.

[0046] The second threshold can be set to 500 in the embodiment of the present application. After grouping and counting the number of jobs under each user (based on "UID"), check whether the total number of jobs reaches the set second threshold (500 jobs in this example). If the number of user jobs is less than or equal to 500, it means that the number of jobs is small, and all jobs of the entire user can be directly striped uniformly without further subdivision, and jump directly to the assumed step S6. If the number of user jobs is greater than 500, it indicates that the number of jobs is large. For the sake of system security and stability, the jobs should be further subdivided according to the "application" to which they belong, and then the jobs under each application should be striped separately, which will lead to step S7.

[0047] The application is the subdirectory of the WorkDir property of the job path for the data.

[0048] The "application" in this application is different from the "application" commonly understood. The "application" in this application is the sub-directories in the "WorkDir" attribute of the data's job path. For example, the job paths of the same user are " / home / alisa / fluent" and " / home / alisa / flowstar", in which "fluent" and "flowstar" are sub-directories, that is, two "applications".

[0049] Step S6, striping the jobs according to the user home directory, and then executing step S9.

[0050] For users who meet certain conditions, all jobs under their home directory are striped, ignoring the influence of the "WorkDir" attribute of a single job path, and adopting a unified strategy to apply striping configuration to all jobs. This simplifies management complexity while ensuring that all jobs of the user can enjoy the performance improvement brought by striping, and then directly execute the striping setting of step S9.

[0051] Step S7: Count the number of jobs under each application and calculate the proportion of jobs that can be striped, and then execute step S8.

[0052] For eligible users, the job records are double-grouped by the "UID" and "Application" fields, and the total number of jobs is calculated for each user-application combination. Within each user-application group, the number of jobs with "type=true", that is, the jobs previously marked as requiring striping, is counted, and then the proportion of these jobs to the total number of jobs under the application is calculated.

[0053] Step S8: Determine the application. If the proportion of the stripe-capable jobs is not less than a third threshold, execute step S9.

[0054] Step S8 also includes: If the proportion of the stripeable jobs is less than the third threshold, the jobs starting with the name of the application in the job path WorkDir are not striped.

[0055] The third threshold can be set to 95% in the embodiment of the present application, and for the users identified in the previous step as needing to be processed by application segmentation, their "application" subdirectories are checked one by one. For each "application" of each user, the proportion of "type=true" (stripable) jobs is calculated again.

[0056] If the proportion of jobs that can be striped under a certain "application" is less than 95%, this indicates that the jobs under the "application" directory do not widely support striping, or the performance improvement brought by striping is limited. Therefore, striping settings will not be performed on the job path "WorkDir" that begins with the name of this "application", and it will remain as it is to avoid unnecessary resource allocation and potential increase in system complexity. If the proportion of jobs in an "application" that can be striped is greater than or equal to 95%, this strongly indicates that most jobs under this "application" are suitable for striping processing, which can significantly improve efficiency. Striping configuration will be performed for the jobs under this "application", that is, step S9 will be executed to perform detailed striping settings.

[0057] Step S9: judging the number of applied jobs, and performing sub-directory striping setting if the number of applied jobs is not less than a fourth threshold.

[0058] The fourth threshold can be set to 100 in the embodiment of the present application. If the number of jobs under a certain "application" is less than 100, it indicates that the number of jobs is relatively small and the impact on the overall system performance is limited. In this case, we will choose not to perform separate sub-directory striping for these "applications", but to maintain the original policy, perform unified striping settings at the user's home directory level, and then jump to the seventh step to continue the subsequent process. If the number of jobs in the "application" is greater than or equal to 100, these "applications" are considered to have a sufficient number of jobs, and their performance optimization deserves more detailed operations. At this point, we will perform sub-directory striping settings for the job path "WorkDir" that begins with the name of these "applications" and perform specific configuration work.

[0059] In one possible implementation, the subdirectory striping is set to set the stripe size to 1 MB and the stripe count to 4.

[0060] After step S9, the method further includes: Step S10: After the job is completed, the stripe setting of the main directory will be restored to the default setting.

[0061] After the job is completed, the directory's striping settings will be restored to the default settings, reducing the potential negative impact on subsequent jobs even if the wrong directory is mistakenly selected.

[0062] In the embodiment of the present application, the main purpose of implementing the statically-aware striping strategy is to refine it according to actual needs and reduce the situation where non-striped jobs are striped. Differentiating striped jobs by user and then by "application" is only one way to achieve this goal. Other division strategies, such as dividing by "application" and then by job name, can also achieve striping of the storage system. Secondly, the thresholds of users, percentage of "applications", and number of jobs are not unique, and each value can be reasonably changed according to the data situation.

[0063] In one possible implementation, based on Figure 2 Please refer to Figure 3 , Figure 3 The static-aware striping optimization framework for jobs proposed in the embodiment of the present application is shown. I / O Trace analysis collects and analyzes existing I / O operation traces (I / OTrace), which record file access patterns in actual workloads. By analyzing this data, the system can identify which jobs or operations can benefit significantly from striping (a technology that distributes data across multiple disks to speed up read and write speeds).

[0064] User job striping potential assessment: Based on the analysis results of I / O Trace, the system further determines the overall characteristics of user jobs, especially the proportion of jobs that can be striped. If most jobs are suitable for striping, it means that the user's job set as a whole can get a significant performance improvement from striping, and the system will tend to enable striping configuration for these users.

[0065] Determine the job number threshold: The system sets a threshold through statistical analysis of the number of user jobs. This threshold is used to distinguish the size of user jobs and adopt different striping strategies accordingly. If the number of jobs is lower than the threshold, it means that the jobs are relatively concentrated, and striping at the user home directory level may be effective enough. On the contrary, if the number of jobs exceeds the threshold, it means that the jobs are widely distributed and require more sophisticated management.

[0066] Striping settings by "application": When the number of user jobs exceeds the threshold, the system enters the "application" processing stage. Here, "application" is defined as a sub-directory under the user directory, which may represent different projects or functional modules. For these sub-directories, the system further evaluates the striping potential of the jobs within them. Only when the striping ratio of the jobs in the sub-directory also meets certain standards will the sub-directory be striped separately. This ensures the accurate implementation of the striping strategy and avoids resource waste.

[0067] Safety considerations and post-job stripe recovery: Throughout the process, the system also has a built-in safety mechanism, that is, after the job is completed, the stripe settings previously made for optimization are automatically restored to the default state. This design reduces the long-term system impact that may be caused by misconfiguration or changes in job characteristics, and increases the reliability and flexibility of the system.

[0068] In summary, the embodiment of the present application applies striping technology to the storage system, and through the exploration and analysis of the job I / O mode, determines the jobs that can be striped and optimized, and can quickly and effectively determine the jobs that can be striped to adapt to the job distribution of different users. At the same time, measures such as the striping recovery mechanism are set to improve security and effectively improve the I / O performance of the storage system.

[0069] Compared with the adaptive file striping technology of the parallel file system in the prior art, the embodiment of the present application has the following beneficial effects: First, the adaptive file striping technology of the parallel file system needs to use a formula to define the level of system load. However, the level of instantaneous system load will be very uneven, while the overall long-term load is balanced. Therefore, it is actually difficult to accurately define the level of system load. The embodiment of the present application bypasses the determination of system load by determining whether a job can be striped, thereby meeting actual needs.

[0070] Second, the embodiment of the present application studies the user and job paths and mines the semantic information therein, because the stripeable jobs are likely to be concentrated in a certain type of "application" of a certain user.

[0071] Third, the embodiments of the present application can set different ratios and thresholds according to the specific system, which is flexible and targeted.

[0072] Fourth, the embodiment of the present application sets up a stripe recovery mechanism to ensure security and stability.

[0073] The following is a possible implementation of a storage system operation static perception stripe optimization device, which is used to execute the various execution steps and corresponding technical effects of the operation static perception stripe optimization method shown in the above embodiment and possible implementation. The device includes: The filtering module is used to analyze offline data and filter out jobs involving write operations through the write_bytes attribute; The summation module is used to count the amount of write I / O data of each job on each OST and sort them in descending order, and sum them up to get the total amount of write I / O data of each job; The marking module is used to judge each job. If 80% of the total write I / O data is concentrated in 20% of the write I / O data in descending order on each OST, an attribute is added to the job and marked as a job to be striped; A first judgment module, configured to execute a second judgment module when the ratio of jobs to be striped to jobs involving write operations is not less than a first threshold; A second judgment module is used to judge whether the number of jobs is less than a second threshold, and if so, execute the striping module; if not, subdivide the jobs, perform striping settings according to applications, and then execute the calculation module; The striping module is used to stripe the jobs according to the user's home directory, and then execute the striping setting module; A calculation module, used for counting the number of jobs under each application and calculating the proportion of jobs that can be striped, and executing a third judgment module; A third judgment module is used to judge the application, and if the proportion of the stripeable jobs is not less than a third threshold, execute the stripe setting module; The striping setting module is used to determine the number of jobs of the application, and perform sub-directory striping setting if the number of jobs of the application is not less than a fourth threshold.

[0074] In a possible implementation, the device further includes a recovery module, which is used to restore the stripe setting of the main directory to the default setting after the job is completed.

[0075] This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the job static-aware strip optimization method provided in the embodiments of the present application. Therefore, the beneficial effects of the job static-aware strip optimization method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0076] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the static job-aware strip optimization method provided in the embodiment of the present application.

[0077] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0078] Since the instructions stored in the storage medium can execute the steps in any job static-aware strip optimization method embodiment provided in the embodiments of the present application, the beneficial effects that can be achieved by any job static-aware strip optimization method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0079] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for optimizing static-aware stripes, characterized in that: The method comprises: Step S1: Analyze offline data and filter out jobs involving write operations through the write_bytes attribute; Step S2: Count the amount of write I / O data of each job on each OST and arrange them in descending order, and sum them to obtain the total amount of write I / O data of each job; Step S3: judge each job. If 80% of the total amount of write I / O data is concentrated in 20% of the write I / O data in descending order on each OST, add an attribute to the job and mark it as a job to be striped. Step S4, executing step S5 when the ratio of the jobs to be striped to the jobs involving write operations is not less than a first threshold; Step S5, determining whether the number of jobs is less than a second threshold, if so, executing step S6, if not, subdividing the jobs, performing striping according to the application, and then executing step S7; Step S6, striping the jobs according to the user home directory, and then executing step S9; Step S7, counting the number of jobs under each application and calculating the proportion of jobs that can be striped, and executing step S8; Step S8, judging the application, if the proportion of the stripeable jobs is not less than the third threshold, executing step S9; Step S9: judging the number of applied jobs, and performing sub-directory striping setting if the number of applied jobs is not less than a fourth threshold.

2. The method for optimizing static-aware stripes according to claim 1, characterized in that: Subdirectory striping is set to set the stripe size to 1MB and the stripe count to 4.

3. The static-aware strip optimization method for jobs according to claim 1, characterized in that: After step S9, the method further includes: Step S10: After the job is completed, the stripe setting of the main directory will be restored to the default setting.

4. The job static-aware strip optimization method according to claim 1, characterized in that: The step S4 further comprises: When the ratio of the jobs to be striped to the jobs involving write operations is less than the first threshold, no striping process is performed.

5. The job static-aware strip optimization method according to claim 1, characterized in that: The application is the subdirectory of the WorkDir property of the job path for the data.

6. The static-aware strip optimization method for jobs according to claim 1, characterized in that: Step S8 also includes: If the proportion of the stripeable jobs is less than the third threshold, the jobs starting with the name of the application in the job path WorkDir are not striped.

7. A static-aware strip optimization device for an operation, characterized in that: The device comprises: The filtering module is used to analyze offline data and filter out jobs involving write operations through the write_bytes attribute; The summation module is used to count the amount of write I / O data of each job on each OST and sort them in descending order, and sum them to obtain the total amount of write I / O data of each job; The marking module is used to judge each job. If 80% of the total write I / O data is concentrated in 20% of the write I / O data in descending order on each OST, an attribute is added to the job and marked as a job to be striped; A first judgment module, configured to execute a second judgment module when the ratio of jobs to be striped to jobs involving write operations is not less than a first threshold; A second judgment module is used to judge whether the number of jobs is less than a second threshold, and if so, execute the striping module; if not, subdivide the jobs, perform striping settings according to applications, and then execute the calculation module; The striping module is used to stripe the jobs according to the user's home directory, and then execute the striping setting module; A calculation module, used for counting the number of jobs under each application and calculating the proportion of jobs that can be striped, and executing a third judgment module; A third judgment module is used to judge the application, and if the proportion of the stripeable jobs is not less than a third threshold, execute the stripe setting module; The striping setting module is used to determine the number of jobs of the application, and perform sub-directory striping setting if the number of jobs of the application is not less than a fourth threshold.

8. The static-aware strip optimization device for operation according to claim 7, characterized in that: The device also includes a recovery module, which is used to restore the stripe setting of the main directory to the default setting after the job is completed.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the job static-aware strip optimization method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the job static-aware strip optimization method according to any one of claims 1 to 6.

Images