A resource scheduling method and a resource scheduling system

Abstract

The present application provides a resource scheduling method, including: collecting operation attribute information of input and output operations of applications on data objects in a storage system during an operation cycle and object attribute information of the data objects, analyzing the input and output operations of the application based on the fine-grained operation attribute information and object attribute information, and obtaining application portrait information. Based on the application portrait information, accurate prediction of the application input and output operations can be achieved. In this way, when executing the same or similar operations, resource scheduling is performed based on the prediction results, which can avoid unnecessary resource occupation and improve resource utilization and application execution efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a resource scheduling method and a resource scheduling system. Background Art

[0002] An application can generally be executed on at least one node. When an application is executed, it generates several processes / threads. From the start to the end of the application execution, most of the processes / threads on all nodes may generate input and output (IO) operations on the underlying storage system. The data set involved in these IO operations is called a work set.

[0003] The IO operations of applications on working sets are generally regular. For example, the IO operations of video playback applications on their working sets are generally sequential reads. By summarizing the IO operation patterns of applications, we can get the IO pattern of the application (i.e., IO pattern). The industry provides a smart cache technology (Smart Cache, SC) to explore IO patterns. Resource scheduling based on the above IO patterns can improve resource utilization.

[0004] Specifically, SC provides a data intelligence service (DIS) component to predict IO patterns. However, the DIS component does not have high prediction accuracy for IO patterns. When scheduling resources based on the IO patterns predicted by the DIS component, on the one hand, the cache hit rate is reduced, resulting in reduced storage resource utilization and reduced application execution efficiency; on the other hand, the predicted computing power is wasted, reducing computing resource utilization. Summary of the invention

[0005] The present application provides a resource scheduling method, which solves the problem of inaccurate IO mode prediction, which leads to reduced resource utilization and reduced application execution efficiency.

[0006] In a first aspect, the present application provides a resource scheduling method. The method can be applied to a resource scheduling system. The resource scheduling system can collect operation attribute information of input and output operations of an application on a data object in a storage system during a job cycle and object attribute information of the data object at a finer granularity. The operation attribute information includes an input and output address, and the object attribute information includes a data object identifier. In this way, the resource scheduling system can accurately profile the application based on the above operation attribute information and object attribute information to obtain application profile information.

[0007] Resource scheduling based on accurate application profile information can improve cache hit rate, thereby improving storage resource utilization and application execution efficiency. This method can also avoid the waste of computing power caused by inaccurate predictions and improve computing resource utilization.

[0008] Optionally, the resource scheduling system may include a computing node, an analysis node, and a scheduling node. The computing node may collect operation attribute information of input and output operations of the application on the data object in the storage system during the operation cycle and object attribute information of the data object. The analysis node may analyze the operation attribute information and the object attribute information to obtain application profile information. The scheduling node may schedule resources according to the application profile information.

[0009] Optionally, at least two of the computing nodes, analysis nodes, and scheduling nodes can be deployed on the same computer device, so that the transmission delay of data such as operation attribute information, data object attribute information, or application profile information can be reduced, thereby improving scheduling efficiency. In addition, the above deployment method can reduce transmission resource overhead and improve resource utilization.

[0010] Optionally, the computing nodes, analysis nodes, and scheduling nodes may also be deployed on different computer devices, respectively. For example, the computing nodes may be deployed on at least one first computer device, the analysis nodes may be deployed on at least one second computer device, and the scheduling nodes may be deployed on at least one third computer device, thereby improving the robustness of the entire resource scheduling system.

[0011] In some possible implementations, the resource scheduling system can schedule storage resources in the following manner. Specifically, the resource scheduling system first converts the application profile information into an information prompt item, and then schedules the storage resources in the storage system according to the information prompt item. For example, according to the continuous reading or skipping information prompted by the information prompt item, the corresponding content of the data object is added to the cache, so as to achieve accurate data pre-fetching, avoid pre-fetching redundant data occupying the cache, and improve the utilization rate of storage resources.

[0012] In some possible implementations, the resource scheduling system converts the application portrait information into an information prompt item in the following manner. Specifically, the resource scheduling system can determine the information prompt template corresponding to the application portrait information, and the information prompt template can be provided by a mode prompt information converter built into the resource scheduling system, and the mode prompt information converter has at least one mode prompt information conversion framework built in, such as a sequential mode prompt information conversion framework, a random mode prompt information conversion framework, and the like, and each mode prompt information conversion framework can provide an information prompt template for a corresponding mode. In this way, the resource scheduling system can determine the information prompt template corresponding to the application portrait information based on the application portrait information combined with the mode prompt information converter, and then obtain the information prompt item based on the field values ​​of each field in the application portrait information combined with the information prompt template.

[0013] In some possible implementations, the resource scheduling system can schedule storage resources of different types of storage systems. For example, the resource scheduling system can schedule resources for storage systems such as Burst buffer, Bee Google filesystem (BeeGFS), or Lustre. Each storage system can support at least one type of command set.

[0014] The resource scheduling system can select a target command type from a command set supported by the storage system according to the information prompt item, and generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item. Then, the command is sent through the target interface of the storage system so that the storage system executes the command to schedule storage resources. In this way, compatibility with different types of storage systems can be achieved, and storage resources of different types of storage systems can be supported for scheduling.

[0015] In some possible implementations, the application profile information includes the distribution of the input and output operations of the application over time. The distribution of the input and output operations over time can be characterized by the operation time period of the input and output operations and the input and output size within the operation time period. Correspondingly, when scheduling resources, the resource scheduling system can schedule computing resources according to the distribution of the input and output operations of the application over time, so that the computing resources of different computer devices can be balanced.

[0016] Optionally, the distribution of input and output operations over time can be presented in the form of a curve graph or a bar graph, where the horizontal axis of the curve graph or the bar graph is the operation time period, and the vertical axis is the input and output size within the operation time period. It should be noted that the operation time period and input and output size presented in the distribution of input and output operations over time are the combined operation time period and input and output size.

[0017] Specifically, the input and output operations for the same data object can be sorted according to the operation time period. If the operation time periods of adjacent input and output operations overlap, the operation time periods of the adjacent operations are merged, and the input and output sizes of the adjacent input and output operations are added together as the input and output size of the input and output operations within the merged operation time period.

[0018] In some possible implementations, the resource scheduling system can determine the input and output operations for the same data object based on the object attribute information, and then combine at least one recognition model based on the operation attribute information of the input and output operations for the same data object. For example, the operation attribute information of the input and output operations for the same data object is input into at least one recognition model, and the recognition result of the recognition model is obtained as application portrait information.

[0019] In some possible implementations, the at least one recognition model includes a first pattern recognition model, which is used to obtain a first input-output pattern of the application in combination with the input-output addresses of the input-output operations for the same data object, and the first input-output pattern includes any one or more of continuity information, sequence information, hotspot information and sequence information, the continuity information is used to identify whether the input-output operation is continuous, the sequence information is used to identify whether the input-output operation is sequential or reverse order, the hotspot information is used to identify a hotspot area of ​​the input-output operation, and the sequence information is used to identify whether the input-output operation is ordered.

[0020] In some possible implementations, the at least one recognition model includes a second pattern recognition model, and the second pattern recognition model is used to combine the input and output addresses of adjacent operations in the input and output operations for the same data object. For example, the input and output addresses of the input and output operations for the same data object are input into the second pattern recognition model, and the second pattern recognition model can automatically determine the input and output addresses of the adjacent operations, and obtain the second input and output pattern of the application based on the input and output addresses of the adjacent operations. The second input and output pattern includes probability information, and the probability information is used to indicate the probability of the second access area being a successor access area of ​​the first access area.

[0021] In some possible implementations, the at least one recognition model includes a third pattern recognition model, and the third pattern recognition model is used to obtain a third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit, and the third input-output pattern includes random information, and the random information is used to identify that the input-output operation is a random input-output operation.

[0022] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one identification model includes a conflict domain identification model, and the conflict domain identification model is used to identify the address overlapping area as a conflict domain when the operation time periods overlap and the input and output addresses overlap.

[0023] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one recognition model includes a time distribution recognition model, and the time distribution recognition model is used to sort the input and output operations according to the operation time period. When adjacent input and output operations have overlapping operation times, the operation time periods of the adjacent input and output operations are merged, and the input and output sizes of the adjacent input and output operations are added to obtain the input and output size of the merged input and output operation. The time distribution of the input and output operations of the application is obtained based on the time period and the input and output size of the input and output operations.

[0024] In some possible implementations, the operation attribute information also includes an operation time period, the object attribute information also includes a survival time period, and the at least one identification model includes a life cycle identification model, and the life cycle identification model is used to obtain the life cycle type of the data object by combining the survival time period and the operation time period.

[0025] In some possible implementations, the data object includes a data object generated when a job is executed, and the at least one recognition model includes a repeatability recognition model, which is used to obtain whether the data object generated when the job is executed has size repeatability in combination with the input and output addresses of the data object generated when the job is executed multiple times.

[0026] In a second aspect, the present application provides a resource scheduling method. The method can be applied to a computing node. Specifically, the computing node collects operation attribute information of the input and output operations of the application on the data object in the storage system during the operation cycle and the object attribute information of the data object. Among them, the operation attribute information includes the input and output address, and the object attribute information includes the data object identifier. Furthermore, the computing node can also store the operation attribute information of the application during the operation cycle and the object attribute information of the data object, and the operation attribute information and the object attribute information are used to perform application profiling on the application and obtain application profiling information.

[0027] In some possible implementations, the operation attribute information also includes any one or more of a thread identifier of a thread to which the input / output operation belongs, a process identifier of a process to which the input / output operation belongs, a node identifier of a node to which the input / output operation belongs, a task identifier of a task to which the input / output operation belongs, and a job identifier to which the input / output operation belongs.

[0028] The computing nodes can also cluster the operation attribute information identified by any one or more of the data object identifier, the thread identifier, the process identifier, the node identifier, the task identifier and the job identifier to obtain the operation attribute information of the data object-level input and output operations, the thread-level input and output operations, the process-level input and output operations, the node-level input and output operations, the task-level input and output operations and / or the job-level input and output operations.

[0029] In some possible implementations, the operation attribute information further includes one or more of an operation type and an operation time period, and the operation time period is represented by a start time and an end time of the input / output operation.

[0030] In some possible implementations, the object attribute information further includes any one or more of a handle and a life span, and the life span is represented by a creation time and a deletion time of the data object.

[0031] In some possible implementations, the computing nodes may also perform clustering according to the operation attribute information identified by the handle to obtain the operation attribute information of the handle-level input and output operations.

[0032] In a third aspect, the present application provides a resource scheduling method. The method can be applied to an analysis node. Specifically, the analysis node obtains the operation attribute information of the input and output operations of the application on the data object in the storage system during the job cycle and the object attribute information of the data object. Among them, the operation attribute information includes the input and output addresses, and the object attribute information includes the data object identifier. Then, the analysis node can generate application portrait information based on the operation attribute information and the object attribute information, and then store the application portrait information of the application. The application portrait information is used to provide information prompts to the scheduling node for scheduling resources when executing the same job.

[0033] In some possible implementations, the analysis node can determine the input and output operations for the same data object based on the object attribute information, and then combine at least one recognition model based on the operation attribute information of the input and output operations for the same data object, such as inputting the operation attribute information of the input and output operations for the same data object into at least one recognition model, and obtaining the recognition result of the recognition model as application portrait information.

[0034] In some possible implementations, at least one recognition model includes a first pattern recognition model. The analysis node can combine the first pattern recognition model according to the input and output addresses of the input and output operations for the same data object, such as inputting the input and output addresses of the input and output operations for the same data object into the first pattern recognition model to obtain the first input and output pattern output by the first pattern recognition model. The first input and output pattern includes any one or more of continuity information, sequence information, hot spot information and sequence information, wherein the continuity information is used to identify whether the input and output operations are continuous, the sequence information is used to identify whether the input and output operations are sequential or reverse, the hot spot information is used to identify the hot spot area of ​​the input and output operations, and the sequence information is used to identify whether the input and output operations are ordered.

[0035] In some possible implementations, at least one recognition model includes a second pattern recognition model. The analysis node may combine the first pattern recognition model with the input and output addresses of adjacent operations in the input and output operations for the same data object, such as inputting the input and output addresses of adjacent operations in the input and output operations for the same data object into the second pattern recognition model to obtain a second input and output pattern output by the second pattern recognition model. The second input and output pattern includes probability information, and the probability information is used to indicate the probability that the second access area is a subsequent access area of ​​the first access area.

[0036] Specifically, for the i-th input / output operation in the input / output operations of the same data object, the second input / output model can determine the first access area according to the input / output address of the i-th input / output operation. For the i+1-th input / output operation in the input / output operations of the same data object, the second pattern recognition model can determine the second access area according to the input / output address of the i+1-th input / output operation. Wherein, i is a positive integer.

[0037] The second pattern recognition model can count the subsequent access areas of the first access area according to the input and output operations for the same data object to obtain a first number, and count the number of times the second access area is accessed after the first access area to obtain a second number. Then, the second pattern recognition model can determine the probability that the second access area is the subsequent access area of ​​the first access area according to the first number and the second number. When the probability reaches a preset probability, the second pattern recognition model identifies the second input and output pattern of the application as a probabilistic pattern.

[0038] In some possible implementations, at least one recognition model includes a third pattern recognition model. The third pattern recognition model is used to obtain a third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit, the third input-output pattern including random information, and the random information is used to identify that the input-output operation is a random input-output operation.

[0039] In some possible implementations, the operation attribute information further includes an operation time period, and the at least one identification model includes a conflict domain identification model, and the conflict domain identification model is used to identify the address overlap area as a conflict domain when the operation time period overlaps and the input and output addresses overlap. The conflict domain can be used to instruct the developer of the application to update the application to resolve the input and output operation conflict.

[0040] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one recognition model includes a time distribution recognition model, and the time distribution recognition model is used to sort the input and output operations according to the operation time period. When adjacent input and output operations have overlapping operation times, the operation time periods of the adjacent input and output operations are merged, and the input and output sizes of the adjacent input and output operations are added to obtain the input and output size of the merged input and output operation. The time distribution of the input and output operations of the application is obtained based on the time period and the input and output size of the input and output operations.

[0041] In some possible implementations, the operation attribute information also includes an operation time period, the object attribute information also includes a survival time period, and the at least one identification model includes a life cycle identification model, and the life cycle identification model is used to obtain the life cycle type of the data object by combining the survival time period and the operation time period.

[0042] In some possible implementations, the data object includes a data object generated when a job is executed, and the at least one recognition model includes a repeatability recognition model, and the repeatability recognition model is used to obtain whether the data object generated when the job is executed has size repeatability in combination with the input and output addresses of the data object generated when the job is executed multiple times. When the data object generated when the job is executed has size repeatability, the stripe can also be configured according to the size of the data object generated when the job is executed.

[0043] In some possible implementations, after the analysis node performs application profiling on the application to obtain application profiling information, the analysis node may also convert the application profiling information into information prompt items so that the scheduling node can schedule storage resources in the storage system according to the information prompt items.

[0044] In some possible implementations, the analysis node may determine an information prompt template corresponding to the application portrait information, and obtain an information prompt item according to field values ​​of each field in the application portrait information in combination with the information prompt template.

[0045] In a fourth aspect, the present application provides a resource scheduling method. The method can be applied to a scheduling node. Specifically, the scheduling node can obtain application portrait information, which is generated based on the operation attribute information of the input and output operations of the application on the data object in the storage system during the operation cycle and the object attribute information of the data object, and then the scheduling node can perform resource scheduling based on the application portrait information.

[0046] In some possible implementations, the scheduling node can obtain information prompt items obtained by converting the application portrait information. The information prompt items are specifically determined by the analysis node to correspond to the information prompt template of the application portrait information, and are obtained according to the field values ​​of each field in the application portrait information combined with the information prompt template. Correspondingly, the scheduling node can schedule storage resources in the storage system according to the information prompt items. For example, the scheduling node can add the corresponding part of the data object to the cache according to the prompt of the information prompt item to achieve accurate data pre-fetching, etc.

[0047] In some possible implementations, the storage system can be of different types. In order to be compatible with different types of storage systems, the scheduling node can select the target command type from the command set supported by the storage system according to the information prompt item, generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item, and send the command through the target interface of the storage system so that the storage system executes the command to schedule storage resources. Based on this, the decoupling of the analysis node and the scheduling node can be achieved. The analysis node only needs to obtain the application portrait information based on the operation attribute information and the object attribute information, and does not need to pay attention to the type of the underlying storage system. The scheduling node can adapt to different types of storage systems by converting the information prompt item into a command that the storage system can understand.

[0048] In some possible implementations, the application profile information includes the distribution of input and output operations of the application over time. The distribution of input and output operations over time is characterized by the operation time period of the input and output operations and the input and output size within the operation time period. When performing resource scheduling, the scheduling node can schedule computing resources according to the distribution of input and output operations of the application over time.

[0049] In a fifth aspect, the present application provides a resource scheduling device, the device comprising:

[0050] A collection module, used to collect operation attribute information of input and output operations of an application on a data object in a storage system during a job cycle and object attribute information of the data object, wherein the operation attribute information includes an input and output address, and the object attribute information includes a data object identifier;

[0051] A storage module is used to store the operation attribute information and the object attribute information, and the operation attribute information and the object attribute information are used to profile the application.

[0052] In some possible implementations, the operation attribute information further includes any one or more of a thread identifier of a thread to which the input / output operation belongs, a process identifier of a process to which the input / output operation belongs, a node identifier of a node to which the input / output operation belongs, a task identifier of a task to which the input / output operation belongs, and a job identifier to which the input / output operation belongs;

[0053] The device also includes:

[0054] A clustering module is used to cluster the operation attribute information identified by any one or more of the data object identifier, the thread identifier, the process identifier, the node identifier, the task identifier and the job identifier to obtain the operation attribute information of the data object level input and output operation, the thread level input and output operation, the process level input and output operation, the node level input and output operation, the task level input and output operation and / or the job level input and output operation.

[0055] In some possible implementations, the operation attribute information further includes one or more of an operation type and an operation time period, and the operation time period is represented by a start time and an end time of the input / output operation.

[0056] In some possible implementations, the object attribute information further includes any one or more of a handle and a life span, and the life span is represented by a creation time and a deletion time of the data object.

[0057] In some possible implementations, the device further includes:

[0058] The clustering module is used to perform clustering according to the operation attribute information identified by the handle to obtain the operation attribute information of the handle-level input and output operations.

[0059] In a sixth aspect, the present application provides a resource scheduling device, the device comprising:

[0060] A communication module, used to obtain operation attribute information of input and output operations of an application on a data object in a storage system during a job cycle and object attribute information of the data object, wherein the operation attribute information includes an input and output address, and the object attribute information includes a data object identifier;

[0061] A generating module, used for obtaining application portrait information according to the operation attribute information and the object attribute information;

[0062] The storage module is used to store the application portrait information, and the application portrait information is used to provide information prompts for scheduling resources of the scheduling node.

[0063] In some possible implementations, the generating module is specifically used to:

[0064] According to the operation attribute information of the input and output operations on the same data object combined with at least one recognition model, the recognition result of the recognition model is obtained as the application portrait information.

[0065] In some possible implementations, the at least one recognition model includes a first pattern recognition model, and the first pattern recognition model is used to obtain a first input-output pattern of the application in combination with the input-output addresses of the input-output operations on the same data object.

[0066] The first input-output mode includes any one or more of continuous information, sequential information, hot spot information and sequence information. The continuous information is used to identify whether the input-output operation is continuous, the sequential information is used to identify whether the input-output operation is sequential or reverse, the hot spot information is used to identify the hot spot area of ​​the input-output operation, and the sequence information is used to identify whether the input-output operation is ordered.

[0067] In some possible implementations, the at least one recognition model includes a second pattern recognition model, and the second pattern recognition model is used to obtain a second input-output pattern of the application in combination with input-output addresses of adjacent operations in the input-output operations for the same data object. The second input-output pattern includes probability information; wherein the probability information is used to indicate the probability that the second access area is a successor access area of ​​the first access area.

[0068] In some possible implementations, the at least one recognition model includes a third pattern recognition model, and the third pattern recognition model is used to obtain a third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit. The third input-output pattern includes random information, and the random information is used to identify that the input-output operation is a random input-output operation.

[0069] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one identification model includes a conflict domain identification model, and the conflict domain identification model is used to identify the address overlapping area as a conflict domain when the operation time periods overlap and the input and output addresses overlap.

[0070] In some possible implementations, the operation attribute information further includes an operation time period, and the at least one recognition model includes a time distribution recognition model. The time distribution recognition model is used to sort the input and output operations according to the operation time period, merge the operation time periods of the adjacent input and output operations when the operation time of adjacent input and output operations overlaps, add the input and output sizes of the adjacent input and output operations to obtain the input and output size of the merged input and output operation, and obtain the distribution of the input and output operations of the application over time according to the time period and the input and output size of the input and output operations.

[0071] In some possible implementations, the operation attribute information further includes an operation time period, and the object attribute information further includes a life time period. The at least one recognition model includes a life cycle recognition model, and the life cycle recognition model is used to obtain the life cycle type of the data object by combining the life time period and the operation time period.

[0072] In some possible implementations, the data object includes a data object generated when a job is executed, and the at least one recognition model includes a repeatability recognition model, which is used to combine the input and output addresses of the data objects generated when the job is executed multiple times to obtain whether the data objects generated when the job is executed have size repeatability.

[0073] In a seventh aspect, the present application further provides a resource scheduling device, the device comprising:

[0074] A communication module, used to obtain application portrait information, wherein the application portrait information is obtained according to operation attribute information of input and output operations of the application on the data object in the storage system during the operation cycle and object attribute information of the data object, wherein the operation attribute information includes input and output addresses, and the object attribute information includes a data object identifier;

[0075] A scheduling module is used to schedule resources according to the application profile information.

[0076] In some possible implementations, the communication module is specifically used to:

[0077] Obtain information prompt items obtained by converting the application portrait information;

[0078] The scheduling module is specifically used for:

[0079] The storage resources in the storage system are scheduled according to the information prompt item.

[0080] In some possible implementations, the scheduling module is specifically used to:

[0081] Selecting a target command type from a command set supported by the storage system according to the information prompt item;

[0082] Generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item;

[0083] The command is sent through the target interface of the storage system so that the storage system executes the command to schedule storage resources.

[0084] In some possible implementations, the application profile information includes the distribution of input and output operations of the application over time, wherein the distribution of input and output operations over time is characterized by the operation time period of the input and output operations and the input and output size within the operation time period.

[0085] Correspondingly, the scheduling module is specifically used for:

[0086] The computing resources are scheduled according to the distribution of the input and output operations of the application over time.

[0087] In an eighth aspect, the present application provides a resource scheduling system, the resource scheduling system comprising at least one first computer device, at least one second computer device, and at least one third computer device;

[0088] The at least one first computer device is used to collect operation attribute information of input and output operations of the application on the data object in the storage system during the operation cycle and object attribute information of the data object, wherein the operation attribute information includes the input and output address, and the object attribute information includes the data object identifier;

[0089] The at least one second computer device is used to obtain application portrait information according to the operation attribute information and the object attribute information;

[0090] The at least one third computer device is used to perform resource scheduling according to the application portrait information.

[0091] Optionally, each computer device includes a processor and a memory, wherein the processor is used to execute instructions in the memory to perform the steps in the above-mentioned resource scheduling method.

[0092] Specifically, the processor of the first computer device is used to execute instructions in the memory of the first computer device to execute the step of collecting operation attribute information of input and output operations of the application on the data object in the storage system during the operation cycle and the object attribute information of the data object. The processor of the second computer device is used to execute instructions in the memory of the second computer device to execute the step of obtaining application profile information according to the operation attribute information and the object attribute information. The processor of the third computer device is used to execute instructions in the memory of the second computer device to execute the step of scheduling resources according to the application profile information.

[0093] In some possible implementations, the at least one second computer device is specifically configured to:

[0094] Converting the application portrait information into information prompt items;

[0095] The at least one third computer device is specifically used for:

[0096] The storage resources in the storage system are scheduled according to the information prompt item.

[0097] In some possible implementations, the at least one second computer device is specifically configured to:

[0098] Determine an information prompt template corresponding to the application portrait information;

[0099] An information prompt item is obtained according to the field value of each field in the application portrait information in combination with the information prompt template.

[0100] In some possible implementations, the at least one third computer device is specifically configured to:

[0101] Selecting a target command type from a command set supported by the storage system according to the information prompt item;

[0102] Generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item;

[0103] The command is sent through the target interface of the storage system so that the storage system executes the command to schedule storage resources.

[0104] In some possible implementations, the application portrait information includes a distribution of input and output operations of the application over time, and the distribution of the input and output operations over time is characterized by an operation time period of the input and output operations and an input and output size within the operation time period;

[0105] The at least one third computer device is specifically used for:

[0106] The computing resources are scheduled according to the distribution of the input and output operations of the application over time.

[0107] In some possible implementations, the at least one third computer device is specifically configured to:

[0108] Determine input and output operations for the same data object according to the object attribute information;

[0109] According to the operation attribute information of the input and output operations on the same data object combined with at least one recognition model, the recognition result of the recognition model is obtained as the application portrait information.

[0110] In some possible implementations, the at least one recognition model includes a first pattern recognition model, which is used to obtain a first input-output pattern of the application in combination with the input-output addresses of the input-output operations for the same data object, and the first input-output pattern includes any one or more of continuity information, sequence information, hotspot information and sequence information, the continuity information is used to identify whether the input-output operation is continuous, the sequence information is used to identify whether the input-output operation is sequential or reverse order, the hotspot information is used to identify a hotspot area of ​​the input-output operation, and the sequence information is used to identify whether the input-output operation is ordered.

[0111] In some possible implementations, the at least one recognition model includes a second pattern recognition model, and the second pattern recognition model is used to obtain a second input-output pattern of the application in combination with the input-output addresses of adjacent operations in the input-output operations on the same data object, and the second input-output pattern includes probability information, and the probability information is used to indicate the probability of the second access area being a successor access area of ​​the first access area.

[0112] In some possible implementations, the at least one recognition model includes a third pattern recognition model, and the third pattern recognition model is used to obtain a third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit, and the third input-output pattern includes random information, and the random information is used to identify that the input-output operation is a random input-output operation.

[0113] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one identification model includes a conflict domain identification model, and the conflict domain identification model is used to identify the address overlapping area as a conflict domain when the operation time periods overlap and the input and output addresses overlap.

[0114] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one recognition model includes a time distribution recognition model, and the time distribution recognition model is used to sort the input and output operations according to the operation time period. When adjacent input and output operations have overlapping operation times, the operation time periods of the adjacent input and output operations are merged, and the input and output sizes of the adjacent input and output operations are added to obtain the input and output size of the merged input and output operation. The time distribution of the input and output operations of the application is obtained based on the time period and the input and output size of the input and output operations.

[0115] In some possible implementations, the operation attribute information also includes an operation time period, the object attribute information also includes a survival time period, and the at least one identification model includes a life cycle identification model, and the life cycle identification model is used to obtain the life cycle type of the data object by combining the survival time period and the operation time period.

[0116] In some possible implementations, the data object includes a data object generated when a job is executed, and the at least one recognition model includes a repeatability recognition model, which is used to obtain whether the data object generated when the job is executed has size repeatability in combination with the input and output addresses of the data object generated when the job is executed multiple times.

[0117] In a ninth aspect, the present application provides a chip, the chip comprising a processor and a chip interface. The chip interface is used to receive instructions and transmit them to the processor. The processor executes the instructions to execute the resource scheduling method described in any one of the first aspect, the second aspect, the third aspect, or the fourth aspect.

[0118] In a tenth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer device, enable the computer device to execute the methods described in the above aspects.

[0119] In an eleventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer device, enables the computer device to execute the methods described in the above aspects.

[0120] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 This is a system architecture diagram of the resource scheduling method in the embodiment of the present application;

[0122] Figure 2This is a schematic diagram of the structure of the first computer device in the embodiment of the present application;

[0123] Figure 3 This is a schematic diagram of the structure of the second computer device in the embodiment of the present application;

[0124] Figure 4 This is a schematic diagram of the structure of a third computer device in an embodiment of the present application;

[0125] Figure 5 This is a schematic diagram of the structure of a computer device in an embodiment of the present application;

[0126] Figure 6 This is a flowchart of the resource scheduling method in an embodiment of the present application;

[0127] Figure 7 A schematic diagram of mode prompt information conversion in an embodiment of the present application;

[0128] Figure 8 This is a schematic diagram of the structure of the scheduler in the embodiment of the present application;

[0129] Fig. 9 This is a schematic diagram of the result of resource scheduling performed by the scheduling node in an embodiment of the present application;

[0130] Fig.10 This is a schematic diagram of the result of resource scheduling performed by the scheduling node in an embodiment of the present application;

[0131] Fig.11 Schematic diagram of the structure of the resource scheduling system in the embodiment of the present application. DETAILED DESCRIPTION

[0132] Applications can be divided into non-distributed and distributed applications according to the operating environment. Non-distributed applications are also called centralized applications. They run in a centralized environment. All functional modules of the application are centrally deployed on the central node, and all application functions are realized through the central node. Distributed applications are applications that run in a distributed environment. They generally include multiple components that run in separate operating environments. These components can collaborate through the network to jointly realize all application functions. In actual applications, distributed applications can be deployed in a cluster formed by multiple computer devices, so that each component of the distributed application can run in a distributed environment formed by different computer devices.

[0133] Computer equipment refers to any device with computing capabilities. In specific implementation, the computer equipment can be a personal computer (PC) such as a mobile phone, tablet computer or desktop computer, or a workstation or server. The computer equipment includes a processor, such as a central processing unit (CPU), a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The computer equipment can provide computing resources for users through the processor.

[0134] A storage system is a system for storing programs and data. The system may include at least one storage device and a control component. Application programs may be stored in the storage system, and the computer device runs the application by reading the program in the storage system. It should be noted that the storage system may be integrated with the computer device or may be independent of the computer device. When the storage system is independent of the computer device, each computer device in the cluster may share the storage system to form a cluster storage system.

[0135] Considering the requirements of read / write performance, capacity, cost, etc., a storage system can generally be configured with multi-level storage devices. Specifically, a storage system can be configured with a cache memory (Cache, referred to as cache), a main memory (Main Memory, also known as internal memory, referred to as memory), and a secondary memory (Secondary Memory, also known as external memory, referred to as external memory).

[0136] The storage system provides different storage resources through the above-mentioned memory. Specifically, the memory can be directly accessed by the processor of the computer device, directly providing data and instructions to the processor, and storing the data processed by the processor. It has a fast read and write speed, but a relatively small capacity; the external memory cannot be directly accessed by the processor, so it is mainly used to store data and programs that the processor does not use temporarily. Its read and write speed is slightly slower than the memory, but its capacity is relatively large; the cache is located between the processor and the memory, and is used to solve the problem of mismatch between the processor's computing speed and the memory's read and write speed. Specifically, the data that the processor will access is stored in the cache. The data is part of the data in the memory. When the processor wants to read the data, it first searches from the cache. If the search is successful, the data is immediately read and sent to the processor for processing. If the search fails, it is read from the memory again.

[0137] In order to improve resource utilization and application execution efficiency during application execution, the industry has proposed a method based on the DIS component to predict data access rules, and then trigger preloading and other means based on the prediction results. However, the above prediction process is based on a short-term time window, and the DIS component is prone to biased weights on the rules and patterns of application data usage, which leads to low prediction accuracy. Especially in complex scenarios, the prediction accuracy of the DIS component is greatly reduced. When scheduling resources, if storage resources such as cache are scheduled according to the above rules and patterns, the cache hit rate may be reduced, which reduces storage resource utilization and application execution efficiency, and wastes the computing power used for prediction, reducing computing resource utilization.

[0138] Based on this, an embodiment of the present application provides a resource scheduling method, which supports fine-grained collection of operation attribute information and object attribute information of the IO operations of the application on the data objects in the storage system during the operation cycle, and performs application profiling on the IO operations of the data objects in the storage system during the entire operation cycle by the application, so as to obtain higher-level and more global application portrait information, and perform resource scheduling based on the global application portrait information, which can improve resource utilization and application execution efficiency.

[0139] Specifically, the application profile information is transmitted to the storage system to guide the storage system to store the working set data more efficiently, improve the cache hit rate, thereby improving the storage resource utilization and application execution efficiency, and avoiding the waste of computing power and improving the utilization of computing resources. In some cases, the computing resources provided by the computer equipment can also be directly scheduled according to the application profile information to improve the utilization of computing resources.

[0140] The resource scheduling method provided in the embodiment of the present application can be applied to, but is not limited to, Figure 1 In the application environment shown.

[0141] like Figure 1 As shown, the cluster 100 includes multiple computing nodes 102, and the multiple computing nodes 102 are connected to the storage system 104. For the job submitted by the user (referred to as a job in this application), the scheduling node 108 can divide the job into multiple tasks (referred to as tasks in this application), and assign the multiple tasks to the multiple computing nodes 102 for execution. When all tasks are executed, it indicates that the job is executed.

[0142] The time period from the start of job execution to the completion of execution can be referred to as a job cycle. During the job cycle, the application running on the computing node 102 generates an application process. The application can generate IO operations on data objects in the storage system 104 based on the application process. Multiple computing nodes 102 can collect the operation attribute information of the IO operations of the application on the computing node on the data objects in the storage system 104 during the job cycle and the object attribute information of the data objects. One of the computing nodes 102 can serve as a summary node to summarize the information collected by multiple computing nodes 102 to obtain the operation attribute information of the IO operations of the application on the data objects in the storage system 104 during the job cycle and the object attribute information of the data objects, and store them in the storage system 104.

[0143] It should be noted that the storage system 104 corresponding to the IO operation generated by the application and the storage system 104 storing the operation attribute information and the object attribute information may be the same storage system or different storage systems. Of course, the aggregation node may also be a node independent of the computing cluster 100. For ease of understanding, this application uses a computing node 102 in the computing cluster 100 as an example of the aggregation node.

[0144] The analysis node 106 may obtain from the storage system 104 the operation attribute information of the IO operation of the application on the data object in the storage system 104 during the operation cycle and the object attribute information of the data object, and perform an application profile on the application according to the operation attribute information and the object attribute information to obtain the application profile information. The analysis node 106 may then store the application profile information of the application, for example, the analysis node 106 may store the above application profile information to the storage system 104.

[0145] In this way, when the scheduling node 108 receives the job again, it can determine whether the job is the same job or a similar job as the job used in the above-mentioned application profiling process based on the job identifier of the job. If so, the corresponding application profiling information can be obtained from the storage system 104 based on the job identifier, and then the scheduling node 108 performs resource scheduling based on the application profiling information. For example, the scheduling node 108 can schedule the computing resources of the computing node based on the distribution of IO operations of the application in the application profiling information over time. For another example, the scheduling node 108 can obtain an information prompt item (referred to as a hint entry in this application) obtained by converting the application profiling information, and schedule the storage resources in the storage system 104 based on the hint entry. In this way, the storage system 104 can accurately obtain data and load it into the cache, thereby realizing accurate data pre-fetching.

[0146] It should be noted that the computing node 102, the analysis node 106 and the scheduling node 108 can be implemented by a computer device. The storage system 104 can be implemented by a computer device with a storage function, or by a storage device such as a hard disk drive (HDD) and / or a solid state drive (SDD). Among them, the computing node 102, the analysis node 106 and the scheduling node 108 can be deployed on the same computer device, so that the transmission overhead can be reduced and the scheduling efficiency can be improved. Of course, the computing node 102, the analysis node 106 and the scheduling node 108 can also be deployed on different computer devices, so that the robustness of the entire resource scheduling system can be improved.

[0147] Figure 1 The example is described by deploying the computing node 102, the analysis node 106 and the scheduling node 108 on different computer devices respectively. In actual application, the above nodes can also be deployed on the same computer device, or partially deployed on the same computer device. Figure 1 The computing node 102 may include multiple computing nodes. In some examples, the computing node 102 may include one computing node.

[0148] For the convenience of description, the present application refers to the computer device that deploys the computing node 102 as the first computer device, the computer device that deploys the analysis node 106 as the second computer device, and the computer device that deploys the scheduling node as the third computer device. When the first computer device has a storage function, such as a processor cache or a non-volatile memory capable of persistent storage, the computing node 102 can also store the operation attribute information and the object attribute information locally, so that the analysis node 106 can directly obtain the operation attribute information and the object attribute information from the computing node 102 for application profiling. Similarly, when the second computer device has a storage function, the analysis node 106 can also store the application portrait information locally, so that the scheduling node 108 can obtain the application portrait information from the analysis node 106 and perform resource scheduling based on the application portrait information.

[0149] Figures 2 to 4 The schematic diagrams of the structures of the first computer device 200, the second computer device 300 and the third computer device 400 are shown respectively. It should be understood that Figures 2 to 4 Only some hardware structures and some software modules in the above-mentioned computer devices are shown. In specific implementation, the first computer device 200, the second computer device 300 and the third computer device 400 may also include more hardware structures, such as speakers, displays, etc., and more software modules, such as various applications, etc.

[0150] like Figure 2As shown, the first computer device 200 includes a bus 201, a processor 202, a communication interface 203 and a memory 204. The processor 202, the memory 204 and the communication interface 203 communicate through the bus 201. The bus 201 can be a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 The communication interface 203 is used to communicate with the outside, for example, to receive operation attribute information and object attribute information of IO operation, etc.

[0151] The processor 202 may be a central processing unit (CPU). The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD or a SSD.

[0152] The memory 204 stores programs or instructions, such as programs or instructions required to implement the functions of the acquisition module, and the processor 202 executes the programs or instructions to execute the aforementioned resource scheduling method. Of course, the memory 204 can also store data, such as data collected by the acquisition module, specifically operation attribute information of IO operations on data objects in the storage system 104 in the operation cycle and object attribute information of the data objects.

[0153] like Figure 3As shown, the second computer device 300 includes a bus 301, a processor 302, a communication interface 303 and a memory 304. The processor 302, the memory 304 and the communication interface 303 communicate with each other through the bus 301. The memory 304 stores programs or instructions, such as programs or instructions required to implement the functions of the analysis module. The analysis module may include at least one recognition model, and correspondingly, the memory 304 stores programs or instructions corresponding to at least one recognition model. The processor 302 executes the program or instruction to execute the aforementioned resource scheduling method. Of course, the memory 304 can also store data, such as data obtained by the analysis module, specifically application portrait information obtained by the analysis module performing application portrait on the application according to the operation attribute information and the object attribute information.

[0154] like Figure 4 As shown, the third computer device 400 includes a bus 401, a processor 402, a communication interface 403 and a memory 404. The processor 402, the memory 404 and the communication interface 403 communicate with each other through the bus 401. The memory 404 stores programs or instructions, such as programs or instructions required to implement the functions of the scheduling module. The processor 402 executes the program or instruction to execute the aforementioned resource scheduling method.

[0155] It should also be noted that the embodiment of the present application also shows a schematic diagram of the structure of the computer device when the computing node 102, the analysis node 106 and the scheduling node 108 are deployed on the same computer device. Figure 5 As shown, the computer device 500 includes a bus 501, a processor 502, a communication interface 503 and a memory 504. The processor 502, the memory 504 and the communication interface 503 communicate with each other through the bus 501. The memory 504 stores programs or instructions, for example, programs or instructions required to implement the functions of the acquisition module, the analysis module and the scheduling module. The processor 502 executes the program or instruction to execute the aforementioned resource scheduling method.

[0156] In some possible implementations, the functions of the computing nodes 102, analysis nodes 106, and scheduling nodes 108 can also be implemented by one or more chips. Each chip includes a processor and a chip interface. The chip interface is used to receive instructions and transmit them to the processor. The processor executes the instructions to perform the steps of the aforementioned resource scheduling method. Of course, when the functions of the computing nodes 102, analysis nodes 106, and scheduling nodes 108 are implemented using multiple chips, the chip interface of each chip receives part of the instructions, and the processor of each chip executes part of the steps of the aforementioned resource scheduling method according to the above-mentioned part of the instructions. In this way, multiple processors can implement all the steps of executing the resource scheduling method.

[0157] In order to make the technical solution of the present application clearer and easier to understand, the resource scheduling method of an embodiment of the present application will be described in detail from the perspective of the interaction between the computing node 102, the storage system 104, the analysis node 106 and the scheduling node 108, taking the computing node 102 deployed in the first computer device 200, the analysis node 106 deployed in the second computer device 300, and the scheduling node 108 deployed in the third computer device 400 as an example.

[0158] See also Figure 6 The resource scheduling method shown is a flowchart, the method comprising:

[0159] S602: The computing node 102 collects operation attribute information of the IO operation performed by the application on the data object in the storage system 104 during the operation cycle and object attribute information of the data object.

[0160] The data object is specifically a set of symbols that can be recognized by the computing node 102 and input to the computing node 102 for processing. Based on different data organization forms, the data object can be a file, a block, or an object. When the application is running, it is often necessary to obtain data objects from the storage system 104 for calculation, and then output the calculation results, or write data objects to the storage system 104, thus generating input and output operations on the data objects, namely IO operations.

[0161] In a specific implementation, the computing node 102 can collect the IO operations of the application on the data objects in the storage system 104 during the operation cycle through an IO recorder residing in the application layer. The IO recorder can be implemented by any IO tracking tool. A computing node 102 in the computing cluster 100 can summarize the information collected by each computing node 102 in the computing cluster 100 to obtain the operation attribute information of the IO operations of the application on the data objects in the storage system 104 during the operation cycle and the object attribute information of the data objects.

[0162] Specifically, for each IO operation applied within the operation cycle, the computing node 102 collects the operation attribute information of the IO operation and the object attribute information of the data object targeted by the IO operation through the IO recorder. Among them, the operation attribute information is used to describe the operation characteristics, and may specifically include the IO address. In a specific implementation, the IO address can be identified by the starting address (i.e., offset) and the IO size (i.e., lens, also referred to as IO Size). In some cases, the IO address can also be identified by the starting address and the ending address. The object attribute information is used to describe the characteristics of the data object, and may specifically include a data object identifier. The data object identifier may be a unique identifier such as the name of the data object.

[0163] In a time period, a data object may be opened by multiple applications, or opened multiple times by one application. In this way, multiple open handles may be generated. The handle may be understood as a pointer used by an application to reference a data object in the storage system 104. In order to distinguish the IO behaviors of different handles, the object attribute information collected by the IO recorder may also include the handle of the data object. Furthermore, after summarizing the information collected by multiple computing nodes 102 in the computing cluster 100, the computing node 102 may also cluster the behavior attribute information according to the handle to obtain the behavior attribute information of the handle-level IO behavior.

[0164] Similarly, in order to distinguish IO operations of different threads, different processes, different nodes, different tasks or different jobs, the operation attribute information collected by the IO recorder can also include any one or more of the thread identifier of the thread to which the IO operation belongs, the process identifier of the process to which the IO operation belongs, the node identifier of the node to which the IO operation belongs, the task identifier of the task to which the IO operation belongs, and the job identifier of the job to which the IO operation belongs.

[0165] Correspondingly, the IO recorder can not only cluster IO operations according to data object identifiers to obtain data object-level IO operations; it can also cluster IO operations according to any one of thread identifiers, process identifiers, node identifiers, task identifiers and job identifiers to obtain any one or more of thread-level IO operations, process-level IO operations, node-level IO operations, task-level IO operations and job-level IO operations.

[0166] For ease of understanding, the following is an explanation with specific examples. For example, if an application generates 10 IO operations in a job cycle, and the thread ID of the thread to which 4 IO operations belong is 0001, the thread ID of the thread to which 3 IO operations belong is 0002, and the thread ID of the thread to which 3 IO operations belong is 0003, then the 4 IO operations with the thread ID 0001 can be grouped into one category, the 3 IO operations with the thread ID 0002 can be grouped into one category, and the 3 IO operations with the thread ID 0003 can be grouped into one category, thereby obtaining thread-level IO operations.

[0167] In some possible implementations, the operation attribute information collected by the IO recorder may also include one or more of an operation type and an operation time period. The operation type specifically refers to the type of IO operation, such as open, create, read, write, or close. The operation time period is characterized by the start time and end time of the IO operation. The operation time period can be used to analyze the distribution of IO operations over time.

[0168] The object attribute information collected by IO recorder can also include a life span. The life span is represented by the creation time and deletion time of the data object. When the data object includes a data object generated during job execution, IO recorder can collect the input and output addresses of the data object generated by multiple job executions to analyze whether the data objects generated during job execution have size repeatability.

[0169] S604: The computing node 102 stores the operation attribute information of the application in the job cycle and the object attribute information of the data object.

[0170] The computing node 102 may store the operation attribute information of the IO operation of the application on the data object in the storage system 104 during the operation cycle and the object attribute information of the data object in the storage system 104. Furthermore, when storing the operation attribute information and the object attribute information, the computing node 102 may also store the corresponding relationship between the operation attribute information and the object attribute information.

[0171] In a specific implementation, the computing node 102 may write the operation attribute information and the object attribute information into an IO log file, and store the IO log file in the storage system 104. It should be noted that the storage system 104 storing the operation attribute information and the object attribute information and the storage system 104 targeted by the IO operation may be the same storage system or different storage systems.

[0172] It should be understood that, in some embodiments, the resource scheduling method may be executed without executing S604.

[0173] S606: The analysis node 106 obtains the operation attribute information of the IO operation performed by the application on the data object in the storage system 104 during the operation cycle and the object attribute information of the data object.

[0174] When the job is completed, the analysis node 106 can obtain the operation attribute information of the application's IO operation on the data objects in the storage system 104 during the job cycle and the object attribute information of the data objects from the storage system 104, so as to perform application profiling based on the operation attribute information and the object attribute information.

[0175] S608: The analysis node 106 performs application profiling on the application according to the operation attribute information and the object attribute information to obtain application profiling information.

[0176] Application portrait refers to describing an application from at least one dimension, and application portrait information refers to information describing an application from at least one dimension. In a specific implementation, the analysis node 106 may describe an application from an IO dimension, based on which the application portrait information may include an IO mode of the application, that is, a mode of accessing data objects by the application.

[0177] Among them, when an application accesses a continuous section of content in a data object, it is called a continuous mode, and when an application accesses discontinuous content in a data object, it is called a non-continuous mode. When an application accesses the content in a data object in order from small to large addresses, it is called a sequential mode, and when an application accesses the content in a data object in order from large to small addresses, it is called a reverse mode. When the frequency of an application accessing the same content in a data object reaches a preset frequency, it is called a hotspot mode. When an application accesses the content in a data object in order, it is called an ordered mode. When the probability of an application accessing the content of one area in a data object and then accessing the content of another area reaches a preset probability, it is called a probabilistic mode. If none of the above modes are hit, the mode in which the application accesses the data object can be determined as a random mode.

[0178] In specific implementation, the analysis node 106 can perform application profiling on the application through at least one recognition model in the analysis module. Specifically, the analysis node 106 can determine the IO operations for the same data object according to the object attribute information, and then input the operation attribute information of the IO operations for the same data object into at least one recognition model, and obtain the recognition result of the recognition model as the application profile information.

[0179] In some possible implementations, the analysis node 106 may input the IO addresses of the IO operations for the same data object into a pattern recognition model, such as a first pattern recognition model, to recognize a first IO pattern of the application. The recognition principle of the first pattern recognition model is to recognize whether the IO pattern is a continuous / non-continuous mode, a sequential mode / reverse mode, a hotspot mode, and / or an ordered mode based on the start address and the end address of the IO operations for the same data object.

[0180] Correspondingly, the first IO mode may include any one or more of continuous information, sequential information, hot spot information and sequence information, wherein the continuous information is used to identify whether the input and output operations are continuous, the sequential information is used to identify whether the input and output operations are sequential or reverse order, the hot spot information is used to identify the hot spot area of ​​the input and output operations, and the sequence information is used to identify whether the input and output operations are ordered.

[0181] The first pattern recognition model can determine whether the IO operation is continuous or discontinuous based on the end address of each IO operation and the starting address of the next IO operation. Specifically, when the end address of each IO operation is equal to the starting address of the next IO operation, the IO operation is determined to be continuous. When the end address of at least one IO operation is not equal to the starting address of the next IO operation, the IO operation is determined to be discontinuous.

[0182] The first pattern recognition model can determine whether the IO operation is sequential or reverse based on the starting address and ending address of each IO operation. Specifically, if the starting address of each IO operation is less than or equal to the ending address, and the ending address of any IO operation is less than or equal to the starting address of the next IO operation, it can be determined that the IO operation is sequential. If the starting address of each IO operation is greater than or equal to the ending address, and the starting address of any IO operation is less than or equal to the ending address of the previous IO operation, it is determined that the IO operation is reversed.

[0183] The first pattern recognition model can also determine the hotspot area of ​​IO operations based on the starting address and ending address of each IO operation. Specifically, for IO operations of the same data object, the analysis node 106 can sort the IO operations according to the IO address (offset), such as sorting in ascending order of offset. If the offset and lens of the IO operation are the same, it indicates that the IO is performed on the same area. The analysis node 106 can deduplicate the same area and accumulate the number of IO operations corresponding to the same area. Furthermore, if there are continuous areas with the same number of times, the continuous areas can also be merged. Then, the analysis node 106 can determine the area with the largest number of IO operations or the area with the highest ranking as the hotspot area.

[0184] The first pattern recognition model can also determine whether the IO operation is orderly based on the starting address and the ending address of each IO operation. Specifically, for the IO operation of the same data object, the first pattern recognition model obtains the sequence of the starting addresses of the IO operation. If the sequence forms an arithmetic sequence, and the sequence of the ending addresses of the IO operation also forms an arithmetic sequence, then the IO operation is determined to be orderly.

[0185] In some possible implementations, the analysis node 106 inputs the IO address of the IO operation for the same data object into a pattern recognition model, for example, a second pattern recognition model, which can analyze the IO addresses of two adjacent IO operations to identify the second IO pattern of the application. The recognition principle of the second pattern recognition model is to determine whether the two adjacent IO operations are regular based on the start address and end address of the two adjacent operations in the IO operation for the same data object. Specifically, it is determined whether the application has a greater probability of accessing another area after accessing a certain area based on the start address and end address of the two adjacent operations.

[0186] Correspondingly, the second input-output mode includes probability information, wherein the probability information is used to indicate the probability that the second access area is a subsequent access area of ​​the first access area, wherein the first access area is an area corresponding to the data object in the previous IO operation, and the second access area is an area corresponding to the data object in the subsequent IO operation.

[0187] For ease of understanding, a detailed description is given below. For the ith IO operation among the IO operations of the same data object, the second pattern recognition model determines the first access area according to the IO address of the ith IO operation. Similarly, for the i+1th IO operation among the IO operations of the same data object, the second pattern recognition model determines the second access area according to the IO address of the i+1th IO operation. Wherein, i is a positive integer.

[0188] The second pattern recognition model can count the successor access areas of the first access area according to the IO operation of the same data object to obtain a first number, and count the number of times the second access area is accessed after the first access area to obtain a second number. Then, the second pattern recognition model can determine the probability of the second access area being the successor access area of ​​the first access area according to the first number and the second number. Specifically, the analysis node 106 can determine the ratio of the second number to the first number as the probability of the second access area being the successor access area of ​​the first access area. When the probability reaches a preset probability, such as 0.8, the second pattern recognition model identifies the second IO pattern of the application as a probabilistic pattern.

[0189] Of course, when neither the first IO pattern nor the second IO pattern is hit, that is, when the IO pattern of the application does not belong to any of the sequential continuous pattern, sequential non-continuous pattern, reverse continuous pattern, reverse non-continuous pattern, hot spot pattern, ordered pattern or probability pattern, the analysis node 106 can identify the third IO pattern of the application through the third pattern recognition model. The third IO pattern includes random information, and the random information is used to identify that the IO operation is a random IO operation.

[0190] In some possible implementations, the operation attribute information may also include the operation time period and / or operation type, etc., and the object attribute information may also include the life span of the data object. Correspondingly, the application profile information may also include any one or more of the conflict domain, the distribution of IO operations over time, the life cycle type of the data object, and the size of the data object generated when the job is executed.

[0191] In some implementations, the analysis node 106 may input the IO address and the operation time period into a conflict domain identification model, and identify the conflict domain through the conflict domain identification model. The so-called conflict domain can be understood as the area where the IO operation conflicts. The identification principle of the conflict domain identification model is that if the operation time periods overlap (including partial overlap or complete overlap), and the IO addresses also overlap, there is a high probability that an IO operation conflict will occur. The conflict domain identification model can identify the area corresponding to the address overlap portion, that is, the address overlap area, as a conflict domain. The conflict domain can be used to instruct the developer of the application to optimize the application.

[0192] In some implementations, the analysis node 106 can input the IO address and the operation time period into the time distribution recognition model, and recognize the time distribution of the IO operations through the time distribution recognition model. Among them, the recognition principle of the time distribution recognition model is to sort the IO operations according to the operation time period. When there is an overlap in the operation time of adjacent IO operations, the operation time periods of the adjacent IO operations are merged, the IO sizes of the adjacent IO operations are added to obtain the IO size of the merged IO operation, and the time distribution of the IO operations of the application is obtained according to the operation time period and the IO size of the IO operation. In other implementations, the analysis node 106 can input the operation time period and the survival time period into the life cycle recognition model, and recognize the life cycle type of the data object through this life cycle recognition model, that is, recognize the length of the life cycle of the data object and determine whether the data object is a temporary data object. Among them, the operation time period can be characterized by the start time and the end time of the IO operation. The survival time period can be characterized by the creation time and the deletion time of the data object. The life cycle recognition model can determine the job start time and the job end time according to the operation time period of the IO operation of the data object in the storage system 104 during the job cycle; determine the life cycle type of the data object according to the job start time, the job end time, the creation time of the data object, and the deletion time of the data object.

[0193] Specifically, the job start time can be denoted as X, the job end time can be denoted as Y, the creation time of the data object is denoted as N, and the deletion time of the data object is denoted as M. When X <= N < M <= Y, it is determined that the life cycle type of the data object is the short life cycle type. That is, the data object is a temporary object.

[0194] In other possible implementations, the data object includes the data object generated during job execution. The analysis node 106 can input the IO addresses of the data objects generated by the job execution multiple times into the repeatability recognition model, and recognize whether the data objects generated during job execution have size repeatability through this repeatability recognition model. Among them, the recognition principle of the repeatability recognition model is specifically to determine the sizes of the data objects generated by the job execution multiple times based on the IO addresses of the data objects generated by the job execution multiple times, and recognize whether the data objects generated during job execution have size repeatability based on the sequence formed by the sizes of the multiple data objects generated by the job execution multiple times.

[0195] In some implementations, if the series tends to converge, it can be determined that the data objects generated when the job is executed have size repeatability. In this case, the repeatability recognition model can output the size of the data object generated when the job is executed as application portrait information. Among them, the size of the data object generated when the job is executed output by the repeatability recognition model can be the convergence value of the above series. In other implementations, if the variance of the series is less than a preset value, it can also be determined that the size of the data object generated when the job is executed has size repeatability. In this case, the size of the data object generated when the job is executed output by the repeatability recognition model can be the average value of the above series. It should be noted that in some possible implementations, the size of the data object generated when the job is executed output by the repeatability recognition model can be the maximum value, minimum value or median value of the above series.

[0196] S610: The analysis node 106 stores application profile information of the application.

[0197] The analysis node 106 may store the application portrait information of the application in the storage system 104. Of course, in some implementations, the analysis node 106 may also store the application portrait information of the application locally. It should be noted that when the analysis node 106 stores the application portrait information, it may also store the correspondence between the job identifier and the application portrait information, so that when the same job is subsequently executed, the application portrait information may be directly obtained based on the correspondence.

[0198] It should be understood that, in some embodiments, the resource scheduling method may be executed without executing S610.

[0199] In the present application, the analysis node 106 analyzes the operation attribute information and the object attribute information in order to profile the application so as to schedule resources based on the application profile information. To this end, the analysis node 106 can also convert the application profile information of the application into an information hint entry so that the scheduling node 108 can perform resource scheduling by reading the information hint entry.

[0200] In specific implementation, the analysis node 106 can determine the information prompt template corresponding to the application portrait information according to the application portrait information. The information prompt template specifically refers to the template of the information prompt item, and the information prompt item can be obtained by filling in the corresponding field values ​​in the template, such as the field values ​​from each field in the application portrait information.

[0201] In some possible implementations, such as when the application profile information includes an IO mode, the analysis node 106 may convert the application profile information into a corresponding hint entry through a mode hint information converter. Figure 7As shown, the mode prompt information converter includes at least one mode prompt information conversion framework. The mode prompt information conversion framework may include one or more of a sequential mode prompt information conversion framework, a random mode prompt information conversion framework, a shared mode prompt information conversion framework, an exclusive mode prompt information conversion framework, or a temporary file (tmpfile) mode prompt information conversion framework, and each mode prompt information conversion framework may provide an information prompt template corresponding to an IO mode. The analysis node 106 may determine the mode prompt information conversion framework corresponding to the application portrait information from at least one mode prompt information conversion framework according to the application portrait information, and then generate a hint entry based on the information prompt template provided by the mode prompt information conversion framework and the field values ​​of each field in the application portrait information.

[0202] It should be noted that an IO mode can be converted into one hint entry or multiple hintentries. Figure 7 As shown, the working set of the application may include three data objects, namely file A, file B and file C, and each data object may have at least one IO mode. For example, there are IO modes 1 and IO mode 2 for file A. The IO mode set formed by the IO modes of each data object is input into the mode hint information converter, and the mode hint information converter can convert the application portrait information based on the information hint template provided by the mode hint information conversion framework to generate the corresponding hint. For example, the two IO modes for file A can be converted into three hint entries, namely hint1, hint2 and hint3, in the above manner, wherein one IO mode is converted into one hint entry and the other IO mode is converted into two hinterries. The conversion process of file B and file C is similar to that of file A, which will not be repeated here.

[0203] Furthermore, the analysis node 106 may also store the above hint entry locally or in the storage system 104, so that when the same or similar jobs are subsequently executed, resources can be scheduled according to the above hint entry to improve resource utilization and application execution efficiency.

[0204] S612: The scheduling node 108 obtains application profile information of the application.

[0205] The scheduling node 108 may directly obtain the application profile information of the application. In some cases, the scheduling node 108 may also obtain a hint entry generated by converting the application profile information.

[0206] S614: The scheduling node 108 performs resource scheduling according to the application profile information.

[0207] Specifically, when the application profile information includes the distribution of the IO operation of the application over time, the scheduling node 108 can directly schedule the computing resources of the computing node 102 according to the distribution of the IO operation of the application over time. By balancing the computing resources of multiple computing nodes 102, the utilization rate of computing resources can be improved.

[0208] Of course, the scheduling node 108 can also provide information prompts to the storage system 104 according to the hint entry, thereby realizing storage resource scheduling. Considering that the storage system 104 may include multiple types, such as Burst buffer, BeeGFS or Lustre, when parsing the hint entry, the scheduling node 108 can select the target command type from the command set supported by the storage system 104 according to the hint entry. For example, the scheduling node 108 can select the target command type willread. Then, the scheduling node 108 generates at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item.

[0209] If the information prompt item indicates that the IO mode is sequential non-continuous (field mode), the starting address (field begin) is 0, the ending address (field end) is 77824, the step length (field stride) is 4096, and the IO size (field block) is 4096, then the scheduling node 108 can generate multiple commands according to the above target command type and the fields associated with the target command type in the information prompt item, such as the field values ​​of mode, begin, end, stride and block. The multiple commands can be expressed as xxx---willread-s 0-e 4096---xxx, xxx---willread-s8192-e12288. Among them, xxx in the command is an omitted field, the first command indicates that the application will read the content with an address range of 0-4096, and the second command indicates that the application will read the content with an address range of 8192-12288.

[0210] The scheduling node 108 also issues the command through the target interface of the storage system 104, thereby realizing information prompting to the storage system 104, and further realizing storage resource scheduling. For example, the scheduling node 108 can issue the above command to enable the storage system 104 to add the content in the address range of 0-4096 and the content in the address range of 8192-12288 to the cache, so that the application can realize accurate data pre-fetching, avoid pre-fetching redundant data occupying the cache, and improve the storage resource utilization.

[0211] In actual application, the scheduling node 108 can implement storage resource scheduling through the scheduler in the scheduling module. Specifically, when the scheduler receives a job, it obtains hintentry from the job description files (JD), and then converts the hint entry into a command corresponding to the storage system 104 by parsing the hint entry content, and then sends the command through the target interface of the storage system 104, thereby delivering the hint entry to the underlying storage system 104. In this way, the storage system 104 can accurately predict the IO operation mode of the application on the data object, and execute the corresponding storage resource scheduling strategy based on the IO operation mode to improve the storage resource utilization and application execution efficiency.

[0212] To facilitate understanding, the following example uses the scheduler as the scheduler to illustrate the storage resource scheduling process.

[0213] See also Figure 8 , scheduler is a scheduler that supports the job callback framework hook framework. The hintentry is specifically stored in the scheduler's job description file JD in the form of a hook record. According to the execution order of the hint entry, for example, before the application's job starts executing or after the execution is completed, the hook record can be divided into prehook and posthook. Each hook record specifies a data object in the working set and the IO operation that is about to occur on the data object. Among them, the data object can be represented by a data object identifier such as a data object name. The IO operation that is about to occur on the data object can be characterized by information including the IO mode. Taking into account the non-continuous read and write situation, the IO operation can be specifically characterized by the IO mode, start address, end address, IO size, step size, etc. Furthermore, the hook record can also specify the node corresponding to the IO operation.

[0214] After the job is submitted to the scheduler, the scheduler reads the job description file JD. When the prehook record is read, the hint notifier is run at the relevant node, such as the node specified by the prehook record, according to the rules in the prehook. The hint notifier can parse the hint through the built-in hint parser, and then generate instructions corresponding to the storage system 104 based on the storage system adapter layer, and send them to each storage system 104 through the target interface of each storage system 104.

[0215] Among them, when the hint entry is a hint entry related to a read operation, the hint notifier can adopt a blocking execution mode and wait for the storage system 104 to return after executing the related operation; when the hint entry is a hint entry related to a write operation, the hint notifier can adopt a non-blocking execution mode and actively query the storage system 104 whether the operation is completed.

[0216] After all prehooks are executed, Schedule starts to schedule jobs to be executed on each computing node. After the job is executed, it continues to read the job description file JD.

[0217] When a posthook record is read, a hint notifer is executed at a relevant node, such as a node specified by the posthook record, according to the rules in the posthook, so as to pass the hint entry to the storage system 104 by way of instructions. For example, a hint notifer can be executed at a relevant node to instruct the storage system 104 to eliminate the working set cache in the cache by eliminating instructions.

[0218] For ease of understanding, the present application also uses a data object as a file to illustrate the storage resource scheduling process.

[0219] In an example, the IO model of the job file test queried by the job user from the database is a sequential non-continuous read model, and its characteristic value is:

[0220] Table 1

[0221] mode 0 begin 0 end 77824 stride 4096 count 10 block 4096

[0222] Among them, mode indicates the type of IO operation. For example, a value of 0 indicates a read operation. begin and end indicate the IO start address and IO end address respectively. stride indicates the stride, count indicates the number of IO slices, and block indicates the size of the IO slice.

[0223] The scheduler can read the job description file JD. When the prehook in the job description file JD is read, the prehook records the information of each field included in the above Table 1. The scheduler can generate a command corresponding to the storage system according to the prehook, and then the scheduler can send a command to the storage system through the target interface of the storage system 104, specifically, hint notifier register--id test--mode 0--begin 0--end 77824--stride 4096--count 10--block 4096. The storage system can schedule storage resources by reading the above command. Fig. 9 As shown, the scheduler can instruct the storage system 104 to add the A2 part of the file to the cache instead of adding the A1 part to the cache, so as to accurately determine the location of the next read operation and accurately prefetch data without prefetching redundant data, thereby improving storage resource utilization and application execution efficiency.

[0224] This example is explained with one data object, and the following example is explained with combination of pre-fetching of multiple data objects in a working set.

[0225] like Fig.10 As shown, the work set includes four data objects, specifically files A to D. The analysis node 106 obtains application portrait information based on the operation attribute information and object attribute information of the IO operation in combination with at least one recognition model, such as the first IO pattern recognition model, the second IO pattern recognition model, and the third IO pattern recognition model. The application portrait information may include information such as the IO mode of the application, such as the sequential read mode for the file. In addition, the operation attribute information may also include a node identifier, and the recognition model may also combine the node identifier of the IO operation for the same file to determine whether the IO mode of the application is a shared mode or an exclusive mode.

[0226] Furthermore, the analysis node 106 can also determine the information prompt template corresponding to the IO mode based on the various mode prompt information conversion frameworks included in the mode prompt information converter, such as the sequential mode prompt information conversion framework, the shared mode prompt information conversion framework, and the exclusive mode prompt information conversion framework, and generate corresponding information prompt items based on the information prompt template and the field values ​​of each field in the IO mode.

[0227] The scheduling node 108 can read the information prompt item, select the target command type from the command set supported by the storage system 104 according to the information prompt item, such as selecting the command type of willread, and then generate at least one command according to the target command type and the field value of the field associated with the above target command type in the information prompt item. Among them, for the target command type of willread, its associated fields may include fields such as sequence, sharing, start address, and end address. The scheduling node 108 can generate at least one command according to the command type of will read and the field values ​​of the fields such as sequence, sharing, start address, and end address.

[0228] Then, the scheduling node 108 transmits the above command to the storage system 104 through the target interface of the storage system 104. Based on the command, the storage system 104 can determine to sequentially share and read file A on node 1, specifically including A2-A4 in file A, to read file C exclusively in reverse order, specifically including C4 to C7 in file C, to sequentially share and read file A on node 2, specifically including A6 in file A, and to sequentially and exclusively skip read file B on node 2, specifically including B2, B4, B6 and B8 in file B, to sequentially share and read file A on node 3, specifically including A8-A10 in file A, and to randomly and exclusively read D5 in file D.

[0229] Based on this, the storage system 104 can preload A2-A4, A6 and A8-A10 into the global cache globalcache, load C7, C6, C5, C4 into the local cache local cache of node A in sequence, load A6 into the localcache of node 2, and load A8, A9 and A10 into the local cache of node 3 in sequence. This can greatly reduce invalid data in the cache, improve cache utilization and hit rate, and improve the performance of the storage system 104 and application execution efficiency.

[0230] In another example, the job user queries the database for the size of the files with repetitive sizes in the job. For example, the size of the files with repetitive sizes in the job is:

[0231] Table 2

[0232] size 8192

[0233] The scheduler can send the file size (i.e., size) to the storage system 104 through the command hint notifier register-id NULL--mode 1--size 8192. The storage system 104 determines the stripe size of the file based on the size. Specifically, when the size is small, a smaller stripe can be set, which can reduce the space wasted by alignment and check blocks, etc.

[0234] Of course, the job user can also query the database whether the file used by the job is a temporary file. For temporary files, you can choose strategies such as reducing redundancy and storing in temporary space. For example, the test file of the job is a temporary file, and its characteristic value is:

[0235] Table 3

[0236] tempfile 1

[0237] The storage system 104 is informed that the test file is a temporary file through the command hint_notifier register-id test --mode 1 --tempfile 1. The storage system 104 can place the test file in the temporary space and cancel the multi-copy strategy or the erasure coding (EC) strategy.

[0238] Combined with the above Figures 1 to 10 , describes in detail the resource scheduling method provided by the present application, and will now describe the resource scheduling system provided by the present application in conjunction with the accompanying drawings.

[0239] See also Fig.11 The schematic diagram of the structure of the resource scheduling system shown in FIG. 1 includes:

[0240] At least one first computer device 200, at least one second computer device 300, and at least one third computer device 400, each computer device comprising a processor and a memory;

[0241] The at least one first computer device 200 is used to collect operation attribute information of input and output operations of an application on a data object in a storage system during a job cycle and object attribute information of the data object, wherein the operation attribute information includes an input and output address, and the object attribute information includes a data object identifier;

[0242] The at least one second computer device 300 is used to obtain application portrait information according to the operation attribute information and the object attribute information;

[0243] The at least one third computer device 400 is used to perform resource scheduling according to the application portrait information.

[0244] The specific implementation of the first computer device 200 can be found in Figure 2 For the description of the related contents of the embodiment shown, the specific implementation of the second computer device 300 can be found in Figure 3 For the description of the related contents of the embodiment shown, the specific implementation of the third computer device 400 can be found in Figure 4 The description of the related contents of the illustrated embodiment will not be repeated here.

[0245] The first computer device 200, the second computer device 300 and the fourth computer device 400 may refer to the following when executing the corresponding method steps. Figure 6 The description of the related contents of the illustrated embodiment will not be repeated here.

[0246] In some possible implementations, the at least one second computer device 300 is used to:

[0247] Converting the application portrait information into information prompt items;

[0248] The at least one third computer device is used for:

[0249] The storage resources in the storage system are scheduled according to the information prompt item.

[0250] In some possible implementations, the at least one second computer device 300 is used to:

[0251] Determine an information prompt template corresponding to the application portrait information;

[0252] An information prompt item is obtained according to the field value of each field in the application portrait information in combination with the information prompt template.

[0253] In some possible implementations, the at least one third computer device 400 is used to:

[0254] Selecting a target command type from a command set supported by the storage system according to the information prompt item;

[0255] Generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item;

[0256] The command is sent through the target interface of the storage system so that the storage system executes the command to schedule storage resources.

[0257] In some possible implementations, the application portrait information includes a distribution of input and output operations of the application over time, and the distribution of the input and output operations over time is characterized by an operation time period of the input and output operations and an input and output size within the operation time period;

[0258] The at least one third computer device 400 is used for:

[0259] The computing resources are scheduled according to the distribution of the input and output operations of the application over time.

[0260] In some possible implementations, the at least one third computer device is used to:

[0261] Determine input and output operations for the same data object according to the object attribute information;

[0262] The operation attribute information of the input and output operations for the same data object is input into at least one recognition model, and the recognition result of the recognition model is obtained as the application portrait information.

[0263] In some possible implementations, the at least one recognition model includes a first pattern recognition model, which is used to obtain a first input-output pattern of the application in combination with the input-output addresses of the input-output operations for the same data object, and the first input-output pattern includes any one or more of continuity information, sequence information, hotspot information and sequence information, the continuity information is used to identify whether the input-output operation is continuous, the sequence information is used to identify whether the input-output operation is sequential or reverse order, the hotspot information is used to identify a hotspot area of ​​the input-output operation, and the sequence information is used to identify whether the input-output operation is ordered.

[0264] In some possible implementations, the at least one recognition model includes a second pattern recognition model, and the second pattern recognition model is used to obtain a second input-output pattern of the application in combination with the input-output addresses of adjacent operations in the input-output operations on the same data object, and the second input-output pattern includes probability information, and the probability information is used to indicate the probability of the second access area being a successor access area of ​​the first access area.

[0265] In some possible implementations, the at least one recognition model includes a third pattern recognition model, and the third pattern recognition model is used to obtain a third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit, and the third input-output pattern includes random information, and the random information is used to identify that the input-output operation is a random input-output operation.

[0266] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one identification model includes a conflict domain identification model, and the conflict domain identification model is used to identify the address overlapping area as a conflict domain when the operation time periods overlap and the input and output addresses overlap.

[0267] In some possible implementations, the operation attribute information also includes an operation time period, and the at least one recognition model includes a time distribution recognition model, and the time distribution recognition model is used to sort the input and output operations according to the operation time period. When adjacent input and output operations have overlapping operation times, the operation time periods of the adjacent input and output operations are merged, and the input and output sizes of the adjacent input and output operations are added to obtain the input and output size of the merged input and output operation. The time distribution of the input and output operations of the application is obtained based on the time period and the input and output size of the input and output operations.

[0268] In some possible implementations, the operation attribute information also includes an operation time period, the object attribute information also includes a survival time period, and the at least one identification model includes a life cycle identification model, and the life cycle identification model is used to obtain the life cycle type of the data object by combining the survival time period and the operation time period.

[0269] In some possible implementations, the data object includes a data object generated when a job is executed, and the at least one recognition model includes a repeatability recognition model, which is used to obtain whether the data object generated when the job is executed has size repeatability in combination with the input and output addresses of the data object generated when the job is executed multiple times.

[0270] It should also be noted that the embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0271] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special circuits.

[0272] However, for this application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of this application can be essentially or partly contributed to the prior art in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, disk or CD, etc., and includes several instructions for enabling a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of this application.

[0273] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0274] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0275] The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can store, or a data storage device such as a training device, data center, etc. that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A resource scheduling method, characterized in that: The method comprises: Collecting operation attribute information of input and output operations of an application on a data object in a storage system during an operation cycle and object attribute information of the data object, wherein the operation cycle includes a time period from the start of the operation to the completion of the operation, the operation is divided into multiple tasks and is assigned to multiple computing nodes for execution, the operation attribute information includes an input and output address and an operation time period, and the object attribute information includes a data object identifier; Obtaining application portrait information according to the operation attribute information and the object attribute information, wherein the application portrait information includes a mode in which an application accesses a data object; Resources are scheduled according to the application profile information.

2. The method according to claim 1, characterized in that The resource scheduling according to the application profile information includes: Converting the application portrait information into information prompt items; The storage resources in the storage system are scheduled according to the information prompt item.

3. The method according to claim 2, characterized in that The converting the application portrait information into an information prompt item includes: Determine an information prompt template corresponding to the application portrait information; An information prompt item is obtained according to the field value of each field in the application portrait information in combination with the information prompt template.

4. The method according to claim 2, characterized in that: The step of scheduling storage resources in the storage system according to the information prompt item includes: Selecting a target command type from a command set supported by the storage system according to the information prompt item; Generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item; The command is sent through the target interface of the storage system so that the storage system executes the command to schedule storage resources.

5. The method according to claim 1, characterized in that The application portrait information includes the distribution of input and output operations of the application over time, and the distribution of the input and output operations over time is characterized by the operation time period of the input and output operations and the input and output size within the operation time period; The resource scheduling according to the application profile information includes: The computing resources are scheduled according to the distribution of the input and output operations of the application over time.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of application portrait information according to the operation attribute information and the object attribute information comprises: Determine input and output operations for the same data object according to the object attribute information; According to the operation attribute information of the input and output operations on the same data object combined with at least one recognition model, the recognition result of the recognition model is obtained as the application portrait information.

7. The method according to claim 6, characterized in that The at least one recognition model includes a first pattern recognition model, which is used to obtain a first input-output pattern of the application in combination with the input-output addresses of the input-output operations for the same data object, and the first input-output pattern includes any one or more of continuity information, sequence information, hot spot information and sequence information; wherein the continuity information is used to identify whether the input-output operation is continuous, the sequence information is used to identify whether the input-output operation is sequential or reverse order, the hot spot information is used to identify a hot spot area of ​​the input-output operation, and the sequence information is used to identify whether the input-output operation is ordered.

8. The method according to claim 7, characterized in that The at least one recognition model includes a second pattern recognition model, and the second pattern recognition model is used to obtain a second input-output pattern of the application in combination with the input-output addresses of adjacent operations in the input-output operations on the same data object, and the second input-output pattern includes probability information; wherein the probability information is used to indicate the probability of the second access area being a successor access area of ​​the first access area.

9. The method according to claim 8, characterized in that The at least one recognition model includes a third pattern recognition model, and the third pattern recognition model is used to obtain the third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit, and the third input-output pattern includes random information, and the random information is used to identify that the input-output operation is a random input-output operation.

10. The method according to claim 6, characterized in that The at least one recognition model comprises a conflict domain recognition model, and the conflict domain recognition model is used to recognize an address overlap region as a conflict domain when the operation time periods overlap and the input and output addresses overlap.

11. The method according to claim 6, characterized in that The at least one recognition model includes a time distribution recognition model, which is used to sort the input and output operations according to the operation time period. When adjacent input and output operations have overlapping operation times, the operation time periods of the adjacent input and output operations are merged, and the input and output sizes of the adjacent input and output operations are added to obtain the input and output size of the merged input and output operation. The time distribution of the input and output operations of the application is obtained based on the time period and the input and output size of the input and output operations.

12. The method according to claim 6, characterized in that The object attribute information also includes a life span, and the at least one identification model includes a life cycle identification model, and the life cycle identification model is used to obtain the life cycle type of the data object by combining the life span and the operation time period.

13. The method according to claim 6, characterized in that The data objects include data objects generated when the job is executed, and the at least one recognition model includes a repeatability recognition model, which is used to combine the input and output addresses of the data objects generated by multiple executions of the job to obtain whether the data objects generated when the job is executed have size repeatability.

14. A resource scheduling system, characterized in that: The resource scheduling system includes at least one first computer device, at least one second computer device and at least one third computer device; The at least one first computer device is used to collect operation attribute information of input and output operations of the application on the data object in the storage system in the operation cycle and object attribute information of the data object, the operation cycle includes the time period from the start of the operation to the completion of the operation, the operation is divided into multiple tasks and is assigned to multiple computing nodes for execution, the operation attribute information includes the input and output address and the operation time period, and the object attribute information includes the data object identifier; The at least one second computer device is used to obtain application portrait information according to the operation attribute information and the object attribute information; The at least one third computer device is used to perform resource scheduling according to the application portrait information.

15. The resource scheduling system according to claim 14, characterized in that: The at least one second computer device is specifically used for: Converting the application portrait information into information prompt items; The at least one third computer device is specifically used for: The storage resources in the storage system are scheduled according to the information prompt item.

16. The resource scheduling system according to claim 15, characterized in that: The at least one second computer device is specifically used for: Determine an information prompt template corresponding to the application portrait information; An information prompt item is obtained according to the field value of each field in the application portrait information in combination with the information prompt template.

17. The resource scheduling system according to claim 15, characterized in that: The at least one third computer device is specifically used for: Selecting a target command type from a command set supported by the storage system according to the information prompt item; Generate at least one command according to the target command type and the field value of the field associated with the target command type in the information prompt item; The command is sent through the target interface of the storage system so that the storage system executes the command to schedule storage resources.

18. The resource scheduling system according to claim 14, characterized in that: The application portrait information includes the distribution of input and output operations of the application over time, and the distribution of the input and output operations over time is characterized by the operation time period of the input and output operations and the input and output size within the operation time period; The at least one third computer device is specifically used for: The computing resources are scheduled according to the distribution of the input and output operations of the application over time.

19. The resource scheduling system according to any one of claims 14 to 18, characterized in that: The at least one second computer device is specifically used for: Determine input and output operations for the same data object according to the object attribute information; According to the operation attribute information of the input and output operations on the same data object combined with at least one recognition model, the recognition result of the recognition model is obtained as the application portrait information.

20. The resource scheduling system according to claim 19, characterized in that: The at least one recognition model includes a first pattern recognition model, which is used to obtain a first input-output pattern of the application in combination with the input-output addresses of the input-output operations for the same data object, and the first input-output pattern includes any one or more of continuity information, sequence information, hot spot information and sequence information, the continuity information is used to identify whether the input-output operation is continuous, the sequence information is used to identify whether the input-output operation is sequential or reverse order, the hot spot information is used to identify a hot spot area of ​​the input-output operation, and the sequence information is used to identify whether the input-output operation is ordered.

21. The resource scheduling system according to claim 20, characterized in that: The at least one recognition model includes a second pattern recognition model, and the second pattern recognition model is used to obtain a second input-output pattern of the application in combination with the input-output addresses of adjacent operations in the input-output operations for the same data object, and the second input-output pattern includes probability information, and the probability information is used to indicate the probability of the second access area being a successor access area of ​​the first access area.

22. The resource scheduling system according to claim 21, characterized in that: The at least one recognition model includes a third pattern recognition model, and the third pattern recognition model is used to obtain the third input-output pattern of the application when neither the first input-output pattern nor the second input-output pattern is hit, and the third input-output pattern includes random information, and the random information is used to identify that the input-output operation is a random input-output operation.

23. The resource scheduling system according to claim 19, characterized in that: The at least one recognition model comprises a conflict domain recognition model, and the conflict domain recognition model is used to recognize an address overlap region as a conflict domain when the operation time periods overlap and the input and output addresses overlap.

24. The resource scheduling system according to claim 19, characterized in that: The at least one recognition model includes a time distribution recognition model, which is used to sort the input and output operations according to the operation time period. When adjacent input and output operations have overlapping operation times, the operation time periods of the adjacent input and output operations are merged, and the input and output sizes of the adjacent input and output operations are added to obtain the input and output size of the merged input and output operation. The time distribution of the input and output operations of the application is obtained based on the time period and the input and output size of the input and output operations.

25. The resource scheduling system according to claim 19, characterized in that: The object attribute information also includes a life span, and the at least one identification model includes a life cycle identification model, and the life cycle identification model is used to obtain the life cycle type of the data object by combining the life span and the operation time period.

26. The resource scheduling system according to claim 19, characterized in that: The data objects include data objects generated when the job is executed, and the at least one recognition model includes a repeatability recognition model, which is used to combine the input and output addresses of the data objects generated by multiple executions of the job to obtain whether the data objects generated when the job is executed have size repeatability.

Images