Resource optimization method and device for heterogeneous computing cluster, electronic equipment and medium

Abstract

The embodiment of the invention discloses a resource optimization method and device for a heterogeneous computing cluster, electronic equipment and a medium, and relates to the technical field of resource optimization. The method comprises the steps of obtaining a first use state of each system resource of each computing node in response to a first processing process of a target task, and obtaining a first real-time resource state data set; under the condition of determining that the allocation of each computing resource of each computing node in the distributed system architecture is unreasonable, the YARN component allocates the computing resources according to the first real-time resource state data set, the computing resource demand of the target task and a task demand model; and under the condition that the allocation of each storage resource of each computing node in the distributed system architecture is determined to be unreasonable, the HDFS component adjusts a replication strategy and / or a storage position of each data block according to the storage resource demand of the target task. According to the scheme provided by the embodiment of the invention, each system resource in the heterogeneous computing cluster can be dynamically adjusted, and the task execution efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of resource optimization, and in particular to a resource optimization method, device, electronic device and medium for a heterogeneous computing cluster. Background Art

[0002] A heterogeneous computing cluster is a high-performance computing environment that combines different types of computing resources, such as central processing units (CPUs) and graphics processing units (GPUs), and other possible accelerators, to meet the needs of complex computing tasks. Heterogeneous computing clusters can provide higher computing power and efficiency than a single type of processor, and are particularly suitable for deep learning, scientific computing, and other tasks that require massive parallel computing.

[0003] In heterogeneous computing clusters, resource scheduling and optimization are key issues to ensure efficient operation of the system. Since heterogeneous computing clusters are composed of different types of computing resources, existing resource scheduling methods often only optimize homogeneous resources and fail to fully consider the characteristics of different computing resources. As a result, in actual operation, some tasks are not reasonably allocated to the most suitable computing nodes, resulting in low resource utilization. In addition, existing task scheduling strategies mainly rely on static resource allocation schemes, lack real-time adjustment capabilities, and cannot cope with dynamic changes in system load.

[0004] During the task processing process, how to dynamically adjust the system resources of the heterogeneous computing cluster to improve the execution efficiency of the task and thus improve the overall performance of the system is a major research issue in the industry. Summary of the invention

[0005] The present invention provides a resource optimization method, device, electronic device and medium for a heterogeneous computing cluster, which can dynamically adjust the system resources of the heterogeneous computing cluster during task processing, improve the execution efficiency of the task, and provide a basis for improving the overall performance of the system.

[0006] According to one aspect of the present invention, a method for optimizing resources of a heterogeneous computing cluster is provided, wherein the heterogeneous computing cluster is deployed in a distributed system architecture, and the method comprises:

[0007] In response to a first processing process of the target task, a first usage status of each system resource of each computing node in the distributed system architecture is obtained to obtain a first real-time resource status data set; the system resources include computing resources and storage resources;

[0008] Determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable;

[0009] In the case where it is determined that the allocation of computing resources of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model;

[0010] When it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task.

[0011] According to another aspect of the present invention, a resource optimization device for a heterogeneous computing cluster is provided. The heterogeneous computing cluster is deployed in a distributed system architecture. The device includes:

[0012] A first real-time resource status data set determination module is used to obtain a first usage status of each system resource of each computing node in the distributed system architecture in response to a first processing process of the target task, and obtain a first real-time resource status data set; the system resources include computing resources and storage resources;

[0013] A determination module, used to determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable;

[0014] A computing resource optimization module, configured to allocate computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model when determining that the computing resource allocation of each computing node in the distributed system architecture is unreasonable;

[0015] The storage resource optimization module is used to adjust the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task when it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the resource optimization method for a heterogeneous computing cluster described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the resource optimization method for a heterogeneous computing cluster described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method for optimizing resources of a heterogeneous computing cluster according to any embodiment of the present invention is implemented.

[0022] The technical solution of the embodiment of the present invention obtains the first usage status of each system resource of each computing node in the distributed system architecture in response to the first processing process of the target task, and obtains a first real-time resource status data set; the system resources include computing resources and storage resources; it is determined whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable; when it is determined that the allocation of each computing resource of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates the computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and the pre-established task requirement model; when it is determined that the allocation of each storage resource of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task, and can dynamically adjust the system resources of the heterogeneous computing cluster during the task processing process, thereby improving the execution efficiency of the task and providing a basis for improving the overall performance of the system.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 is a flow chart of a resource optimization method for a heterogeneous computing cluster provided according to Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of a resource optimization method for a heterogeneous computing cluster provided according to Embodiment 2 of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a resource optimization device for a heterogeneous computing cluster provided in Embodiment 3 of the present invention;

[0028] Figure 4 It is a structural schematic diagram of an electronic device for implementing the resource optimization method of a heterogeneous computing cluster according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] The acquisition, storage, use, and processing of data in the present invention are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present invention, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present invention, but it does not mean that the applicant has or will necessarily use the relevant content of the solution.

[0032] Embodiment 1

[0033] Figure 1This is a flowchart of a resource optimization method for a heterogeneous computing cluster provided according to Example 1 of the present invention. This embodiment can be applied to the situation where resources of a heterogeneous computing cluster deployed in a distributed system architecture are optimized during task processing. The method can be executed by a resource optimization device for a heterogeneous computing cluster. The resource optimization device for a heterogeneous computing cluster can be implemented in the form of hardware and / or software. The resource optimization device for a heterogeneous computing cluster can be configured in each computing node (for example, a computer or electronic device such as a server) in the distributed system architecture.

[0034] Before introducing the specific steps of this embodiment, a brief introduction is given to the heterogeneous computing cluster involved in this embodiment; illustratively, a set of heterogeneous computing clusters is deployed in a cloud computing data center (distributed system architecture), and the heterogeneous computing cluster includes 100 computing nodes, including 50 CPU nodes, 30 GPU nodes, and 20 FPGA nodes; the heterogeneous computing cluster can handle complex tasks from different customers, such as video processing tasks, image recognition tasks, or big data analysis tasks, which are not limited in this embodiment. It should be noted that the distributed system architecture involved in this embodiment can be a Hadoop distributed system architecture or other artificial intelligence distributed system architecture, which is not limited in this embodiment.

[0035] Specifically, refer to Figure 1 , the method comprising:

[0036] Step 110 , in response to the first processing procedure of the target task, obtain a first usage status of each system resource of each computing node in the distributed system architecture to obtain a first real-time resource status data set.

[0037] Among them, the system resources include computing resources and storage resources; illustratively, the computing resources can be CPU utilization or memory utilization, etc.; the storage resources can be disk input / output utilization or network bandwidth occupancy, etc., which are not limited in this embodiment.

[0038] In this embodiment, the target task may be a processing task from different users; for example, a video processing task, an image recognition task, a data mining task, or an abnormal data analysis task, etc., which is not limited in this embodiment.

[0039] It can be understood that the first processing process of the target task can be determined by the start time of the target task. Exemplarily, the first processing process can be a smaller time period after the start of execution of the target task, for example, the tenth minute or the first hour after the start of execution of the target task; the first processing process of the target task can also be determined by the processing progress of the target task. For example, the stage from the start of processing of the target task to the completion of one-third of the target task is determined as the first processing process, which is not limited in this embodiment.

[0040] In an optional implementation of this embodiment, when it is determined that the target task has been processed to the first processing process, the current usage status of each system resource of each computing node in the distributed system architecture can be further obtained, which is referred to as the first usage status in this embodiment, and the first real-time resource status data set is obtained.

[0041] Optionally, in this embodiment, when it is determined that the target task is processed to the first processing process, the system resource status data of each computing node in the distributed system architecture may be collected, wherein the system resource status data may include CPU utilization C i 、Memory usage M i , Disk I / O utilization D i Or the network bandwidth usage N i , where i represents the i-th computing node. Further, a first real-time resource status data set R can be established based on the collected resource status data. i (t), represents the overall resource utilization status of the i-th computing node at time t: R i (t) = f(C i (t), M i (t), D i (t), N i (t)); where f() is the function expression of resource status.

[0042] In another optional implementation of this embodiment, the first real-time resource status data set may be constructed based on the following constraints:

[0043] 0≤C i (t), M i (t), D i (t), N i (t)≤1;

[0044] Among them, CPU utilization C i 、Memory usage M i , Disk I / O utilization D i The value range of the network bandwidth occupancy rate is [0,1], which can indicate that the utilization rate of the corresponding resource is between 0 and 100%.

[0045] Step 120: Determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable.

[0046] In an optional implementation of the present embodiment, after obtaining the first real-time resource status data set, it is possible to further determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable based on the first real-time resource status data set; it can be understood that whether each system resource is reasonable may include: both computing resources and storage resources are allocated reasonably, both computing resources and storage resources are allocated unreasonably, computing resources are allocated reasonably but storage resources are allocated unreasonably, or computing resources are allocated unreasonably but storage resources are allocated reasonably, and the specific circumstances are not limited in the present embodiment.

[0047] Optionally, in this embodiment, determining whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable may include: inputting the first real-time resource status data set into the YARN component of the distributed system architecture; the YARN component determines whether the allocation of each system resource of each computing node in the current distributed system architecture is reasonable based on the first real-time resource status data set and / or the task requirements of the target task; the task requirements include: computing resource requirements and / or storage resource requirements.

[0048] In an optional implementation of this embodiment, after receiving the first real-time resource status data set, the YARN component can determine whether the system resource allocation of each computing node in the current distributed system architecture is reasonable based on the first real-time resource status data set; it can also determine whether the system resource allocation of each computing node in the current distributed system architecture is reasonable based on the task requirements of the target task; it can also determine whether the system resource allocation of each computing node in the current distributed system architecture is reasonable based on the first real-time resource status data set and the task requirements of the target task, which is not limited in this embodiment.

[0049] For example, in an example of this embodiment, after receiving the first real-time resource status data set, if the YARN component determines that the CPU utilization in the first real-time resource status data set reaches 80%, but the GPU utilization is only 20%, then it can be determined that the allocation of computing resources of each computing node in the current distributed system architecture is unreasonable.

[0050] Illustratively, in another example of the present embodiment, after receiving the first real-time resource status data set, the YARN component determines that some data blocks are allocated to storage nodes that are far away from the target computing node based on the first real-time resource status data set and the storage resource requirements in the task requirements of the target task, resulting in excessively high cross-node data transmission delay. At this time, it can be determined that the storage resource allocation of each computing node in the current distributed system architecture is unreasonable.

[0051] In an optional implementation of this embodiment, the first real-time resource status data set R i (t) The data is input to the YARN component for analysis. The YARN component can determine whether the allocation of system resources of each computing node in the distributed system architecture is reasonable based on the utilization rate of each type of resource in the first real-time resource status data set and the computing resource demand or storage resource demand of the target task, thereby providing a basis for dynamically adjusting the matching strategy between the target task and the system resources.

[0052] Step 130: When it is determined that the allocation of computing resources of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model.

[0053] In an optional implementation of this embodiment, when it is determined based on the YARN component that the allocation of computing resources of each computing node in the distributed system architecture is unreasonable, the YARN component can further allocate computing resources in the distributed system architecture according to the computing resource requirements in the task requirements of the target task of the first real-time resource status data set and a pre-established task requirement model.

[0054] Optionally, in this embodiment, the task requirement model can be determined by the following steps: determining the total resource requirement of the target task based on the task load information required for the target task; the task load information includes: the computing power, data transmission rate and storage requirements required to complete the target task; determining the target task requirement model of the target task based on the total resource requirement of the target task; determining the first comprehensive resource requirement of the Map phase and the second comprehensive resource requirement of the Reduce phase in the MapReduce task; the first comprehensive resource requirement includes the resource requirement required for all tasks in the distributed system architecture in the Map phase; the second comprehensive resource requirement includes the resource requirement required for all tasks in the distributed system architecture in the Reduce phase; determining the comprehensive requirement model corresponding to the distributed system architecture based on the first comprehensive resource requirement and the second comprehensive resource requirement.

[0055] In an optional implementation of this embodiment, the total resource requirement of the target task may be determined based on the computing power, data transmission rate, and storage requirement required to complete the target task; illustratively, the total resource requirement of different tasks may be determined according to the following formula:

[0056] L k (t) = α k ·P k (t)+β k ·T k (t)+γ k ·S k (t);

[0057] Among them, L k (t) is the total resource requirement of the kth task at time t; P k (t) represents the computing power required to complete the kth task; T k (t) represents the data transmission rate required to complete the kth task; S k (t) represents the storage requirement required to complete the kth task; α k , β k and γ k They represent the weight factors of the kth task on computing power, data transmission rate and storage requirements respectively.

[0058] Furthermore, the target task requirement model of the target task can be determined according to the total resource requirement of the target task; the target task requirement model of the target task can be expressed as:

[0059]

[0060] Among them, λ k , μ k 、v k are the time decay factors of the target task on different resources respectively.

[0061] Furthermore, the first comprehensive resource requirement of the Map phase and the second comprehensive resource requirement of the Reduce phase in the MapReduce task in the distributed system architecture can be determined; wherein the first comprehensive resource requirement includes the resource requirement required for all tasks in the distributed system architecture in the Map phase; and the second comprehensive resource requirement includes the resource requirement required for all tasks in the distributed system architecture in the Reduce phase.

[0062] In this embodiment, the first comprehensive resource requirement and the second comprehensive resource requirement can be respectively expressed as:

[0063]

[0064] Among them, M m (t) is the first comprehensive resource demand, M r (t) is the second comprehensive resource demand, N m 、N r are the number of nodes for Map and Reduce tasks respectively, α mi , β mi , γ mi and α rj , β rj , γ rj is the weight coefficient of the task in different resource dimensions.

[0065] Furthermore, a comprehensive requirement model corresponding to the distributed system architecture may be determined according to the first comprehensive resource requirement and the second comprehensive resource requirement, and the comprehensive requirement model is the task requirement model involved in this embodiment.

[0066] Optionally, in this embodiment, the first comprehensive resource demand and the second comprehensive resource demand may be integrated in time series, and a time adjustment parameter may be introduced to dynamically control the balanced allocation of resources between the Map stage and the Reduce stage, thereby forming a comprehensive demand model; illustratively, the comprehensive demand model formed in this embodiment may be:

[0067]

[0068] Among them, θ m ,θ r It is the time adjustment parameter of Map and Reduce phases, τ0 and τ1 are the starting time of these two phases.

[0069] In an optional implementation of this embodiment, after obtaining the comprehensive demand model, that is, the task demand model involved in this embodiment, the comprehensive demand model M can be used to MapReduce (t) Combined with the first real-time resource status data set R i (t) Optimize the task allocation strategy of the Map phase and the Reduce phase in the MapReduce task. Specifically, the optimal resource allocation of each computing node in the heterogeneous computing cluster can be achieved by minimizing the difference between resource utilization and allocation ratio:

[0070]

[0071] Among them, x i (t) is the resource allocation ratio of the i-th computing node at time t, R mi (t) and R ri (t) represents the resource utilization of Map and Reduce tasks on the i-th node, respectively, and w m and w ris the weighting coefficient of the task phase.

[0072] Optionally, in this embodiment, the YARN component allocates computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model, which may include: determining the adaptability scoring function of each task in each computing resource of the distributed system architecture according to the task requirement model of each task in the distributed system architecture and the first real-time resource status data set; determining the adaptability score of each task in each computing node according to the adaptability scoring function, and sorting each of the adaptability scores in a set order; determining at least one target computing node according to the sorting result, and each of the target computing nodes is used to run the target task.

[0073] In an optional implementation of this embodiment, the YARN component can be configured according to the task requirement model M k (t) and the first real-time resource status data set R i (t) Define the adaptability scoring function A of the task on different computing resources k，i (t):

[0074]

[0075] Among them, A k，i (t) represents the fitness score of task k on the i-th computing node, w k is the weight factor of the task, indicating the degree of dependence of the task on computing resources, M k (t) is the task requirement model, R i (t) is the first real-time resource status data set.

[0076] Furthermore, the fitness scores of all tasks k on each node i can be compared, and the computing node i with the highest fitness score can be selected. opt :

[0077]

[0078] Where N is the set of all computing nodes in the heterogeneous computing cluster, i opt is the optimal computing node, that is, the target computing node involved in this embodiment, used to execute task k (for example, the target task).

[0079] Step 140: When it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task.

[0080] In an optional implementation of this embodiment, when it is determined that the storage resource allocation of each computing node in the distributed system architecture is reasonable, the HDFS component in the distributed system architecture can adjust the replication strategy and / or storage location of each data block according to the storage resource requirements of the target task.

[0081] In this embodiment, the storage resource requirements include: input / output utilization requirements; the input / output utilization requirements include: read / write speed requirements and data block size requirements.

[0082] Accordingly, the HDFS component adjusts the replication strategy and storage location of each data block according to the storage resource requirements in the task requirements of the target task, which may include: the HDFS component determines the replication strategy according to the read and write speed requirements and the data block size requirements; and / or determines the storage location of each target data block that matches the target task according to the read and write speed requirements of the target task, the computing power of each computing node, and the network delay between the target task and each computing node.

[0083] Optionally, in this embodiment, a storage optimization strategy based on task load is introduced in the HDFS component to collect the input / output utilization requirements of the task in real time. The input / output utilization requirements include the read and write speed required by the task. k (t) and data block size B k (t); Read and write speed I according to the input / output utilization requirements of the task k (t) and data block size B k (t) Dynamically adjust the replication strategy of data blocks: Exemplarily, the replication strategy of data blocks can be dynamically adjusted according to the following formula:

[0084]

[0085] Among them, R k (t) is the number of data blocks copied by the kth task at time t, n k The basic replication factor of the task is used to dynamically adjust the number of copies of the data block based on the I / O requirements of the task and the size of the data block.

[0086] Furthermore, the storage location of data blocks can be dynamically adjusted by analyzing the computing requirements and I / O requirements of the task:

[0087]

[0088] Among them, L k (t) is the optimal storage location of the data block for the kth task at time t, d k，j (t) represents the network delay between the kth task and the jth computing node, Cj (t) is the computing capacity of the jth computing node.

[0089] Furthermore, the distribution of data blocks in the cluster can be dynamically adjusted based on the computing requirements and input / output utilization requirements of the analysis task, so that the data blocks are stored on the nodes to be calculated. For example, the specific optimization goal can be to minimize the data transmission delay:

[0090]

[0091] Among them, O storage (t) is the optimization target of data block storage at time t, K is the total number of tasks in the cluster, and data transmission delay and I / O efficiency work together to optimize the storage location of data blocks.

[0092] In this embodiment, the replication strategy and storage location of the data block can be adjusted in real time. When the load changes, the data is stored on the optimal node and dynamically updated in combination with the computing requirements and input / output utilization requirements to optimize the storage and computing collaborative efficiency of the heterogeneous computing cluster.

[0093] The scheme of this embodiment obtains the first usage status of each system resource of each computing node in the distributed system architecture in response to the first processing process of the target task, and obtains a first real-time resource status data set; the system resources include computing resources and storage resources; it is determined whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable; when it is determined that the allocation of each computing resource of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates the computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and the pre-established task requirement model; when it is determined that the allocation of each storage resource of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task, and can dynamically adjust the system resources of the heterogeneous computing cluster during the task processing process, thereby improving the execution efficiency of the task and providing a basis for improving the overall performance of the system.

[0094] On the basis of the above technical solution, in an optional implementation of this embodiment, the execution strategy of each task in the heterogeneous computing cluster can be dynamically adjusted. For example, the computing resources and storage resources of the heterogeneous computing cluster can be reallocated according to the adaptability score and resource allocation of each task, so that each task can run in the most suitable computing node.

[0095] In this embodiment, the overall performance of the system can be maximized by optimizing the collaborative efficiency of computing resources and storage resources of the heterogeneous computing cluster. Exemplarily, the collaborative efficiency of computing resources and storage resources of the heterogeneous computing cluster can be optimized by the following formula:

[0096]

[0097] Among them, O performance (t) is the optimization target of the overall performance of the heterogeneous computing cluster, K is the total number of tasks in the cluster, and the overall resource utilization efficiency of the system is optimized by maximizing the adaptability score of the task.

[0098] The solution of this embodiment can dynamically adjust the resource scheduling strategy according to the real-time load changes of the heterogeneous computing cluster, always maintain the best match between computing resources and storage resources during task execution, maximize the overall performance of the system, and provide real-time feedback of the scheduling results through the YARN resource scheduler.

[0099] Embodiment 2

[0100] Figure 2 1 is a flow chart of a resource optimization method for a heterogeneous computing cluster provided according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution. The technical solution in this embodiment can be combined with each optional solution in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0101] Step 210: In response to the first processing process of the target task, a first usage status of each system resource of each computing node in the distributed system architecture is obtained to obtain a first real-time resource status data set.

[0102] In an optional implementation of this embodiment, before responding to the first processing process of the target task, if a start processing instruction of the target task is received, the target task can be allocated through the YARN component of the distributed system architecture based on the task requirements of the target task and the load conditions of each computing node in the current distributed system architecture. For example, 10 computing nodes can be determined to process the target task.

[0103] Step 220: Determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable.

[0104] Step 230: When it is determined that the allocation of computing resources of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model.

[0105] Step 240: When it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task.

[0106] Step 250, in response to the continuous processing instruction of the target task, continuously obtain the second usage status of each system resource of each computing node in the distributed system architecture, obtain a second real-time resource status data set, and the execution progress of the target task; determine the load change function of the target task according to the second real-time resource status data set and the execution progress of the target task; adjust the execution strategy of the target task according to the load change function.

[0107] In an optional implementation of the present embodiment, after the first processing process response to the target task is completed, if continuous processing instructions for the target task continue to be received, the current usage status of each system resource of each computing node in the distributed system architecture can continue to be obtained, which is referred to as the second usage status in the present embodiment, thereby obtaining a second real-time resource status data set and the current execution progress of the target task; further, the load change function of the target task can be determined based on the execution progress of the target task in the second real-time resource status data set; finally, the execution strategy of the target task can be adjusted according to the load change function.

[0108] Optionally, in this embodiment, during the continuous execution of the target task, the second resource status data R of each computing node may be continuously collected. i (t) and task execution progress E k (t), where E k The value range of (t) is 0 to 1, indicating the state of the task from never starting to completion; further, the task load change function L can be defined based on the collected resource status data and task execution progress. k (t), the load change function can be used to monitor the system load in real time:

[0109]

[0110] Among them, L k (t) is the load change of the kth task at time t, which is used to measure the matching degree between task execution progress and resource utilization.

[0111] Furthermore, based on the task load change function L k (t) Adjust the execution strategy of the task in real time. For example, when the task load changes beyond the preset threshold L thresholdAt the same time, the computing resources and storage resources can be reallocated according to the second resource status data R i (t) and the input / output requirements of the target task, dynamically adjust the storage location of the data block so that the data block is close to the node where the calculation is to be performed to optimize the data transmission path. The specific optimization goal is to minimize the data transmission delay:

[0112]

[0113] Among them, B k (t) is the data block size of task k, I k (t) is the I / O demand, d k，j (t) is the network delay between the kth task and the jth node, and K is the total number of tasks.

[0114] Step 260 : In response to the execution completion instruction of the target task, the execution data of the target task is stored, and the scheduling library in the YARN component is updated based on the execution data.

[0115] The scheduling library stores execution data of different tasks, and the execution data is used to provide a basis for formulating execution strategies for new tasks.

[0116] In an optional implementation of this embodiment, after receiving the target task processing completion instruction, various execution data of the target task during the execution process may be stored, and the scheduling library in the YARN component may be updated based on the execution data.

[0117] Optionally, in this embodiment, after the target task is completed, the data during the execution of the target task is archived and analyzed to generate optimization feedback information, and used to update the scheduling policy library in the YARN component. Through the closed-loop feedback mechanism, the task execution strategy is effectively optimized to ensure that future tasks can more accurately match computing resources and storage resources when allocating resources, thereby achieving adaptive dynamic optimization of resource scheduling.

[0118] The solution of this embodiment can continuously monitor the progress of task execution and resource utilization during task execution, adjust the task scheduling strategy in real time, so that the system can maintain efficient operation when the load fluctuates, and dynamically adjust the coordinated allocation of storage and computing resources to prevent task execution from causing performance degradation due to resource mismatch; by continuously monitoring the system's resource status and task progress, and combining the feedback mechanism of the YARN scheduler to optimize task allocation and resource utilization strategies, the system can still maintain the preset operating efficiency when the load changes greatly.

[0119] The solution of this embodiment constructs a comprehensive real-time resource status data set by dynamically monitoring the utilization rate, memory usage, disk utilization, and network bandwidth occupancy rate of each computing node in a heterogeneous computing cluster; through the resource scheduler combined with the task demand model, dynamic matching of tasks and computing resources is achieved, effectively solving the problem of mismatch between task execution and resource utilization in heterogeneous clusters, and significantly improving the overall performance of the system. In view of the differences between the Map phase and the Reduce phase of the MapReduce task, the I / O requirements, computing requirements, and data block distribution of the task are collected in real time, and the replication strategy and storage location of the data block are dynamically adjusted to achieve the optimal matching between the data block and the computing node, greatly reducing the data transmission delay and effectively improving the overall efficiency of task execution; by introducing an adaptability scoring function based on the task demand model and combining it with the real-time resource status data set, the highest scoring node is selected for task execution according to the adaptability score of each task on different computing nodes, and the allocation strategy of computing resources and storage resources is dynamically optimized through the feedback mechanism of the YARN component to ensure that the system remains stable when the load changes. It can maintain efficient operation; by adjusting the task execution strategy, data block storage location and resource allocation in real time, it can effectively cope with system load fluctuations, ensure maximum resource utilization during task execution, and ensure flexible adjustment of task execution strategy through task load change function, which significantly improves the stability and efficiency of heterogeneous computing clusters; after the task is completed, it generates optimization feedback information by archiving and analyzing the data during the task execution process, and uses it to update the scheduling strategy library in the YARN component. Through the closed-loop feedback mechanism, it effectively optimizes the task execution strategy, ensures that future tasks can more accurately match computing resources and storage resources when allocating resources, and realizes adaptive dynamic optimization of resource scheduling.

[0120] In order to better understand the resource optimization method of the heterogeneous computing cluster involved in this embodiment, a specific example is used below to illustrate it:

[0121] In this embodiment, a heterogeneous computing cluster is deployed in a cloud computing data center, which includes 100 computing nodes, consisting of 50 CPU nodes, 30 GPU nodes and 20 FPGA nodes. The data center is mainly responsible for processing complex tasks from different customers, including applications such as video processing, image recognition and big data analysis.

[0122] In the specific implementation, we first selected a task scenario involving massive image processing. The task is to obtain real-time video stream from a camera system and perform image recognition processing on the video frames. The data set is 1TB of image data. The task is to perform real-time recognition of the data and generate statistical reports. The task requirements are highly dependent on the parallel computing capabilities of the GPU, so we need to ensure that the GPU resources are fully utilized.

[0123] Within the first hour after the task is started, the resource status of all computing nodes is monitored in real time. The collected data shows that the average utilization of GPU nodes is only 40%, while the utilization of CPU nodes is close to 90%, indicating that GPU resources are not fully utilized and CPUs are overloaded. At this time, the YARN component reallocates computing resources based on the acquired real-time resource status data set, the task requirements of the task, and the pre-established task requirement model, and transfers some tasks from CPU nodes to GPU nodes. After the adjustment, the average GPU utilization rate rises to 75%, while the CPU utilization rate drops to 70%, and the overall performance of the cluster is significantly improved. In the next 4 hours, the system continuously monitors the task execution progress and resource status, and dynamically adjusts task allocation according to load fluctuations to ensure that tasks always run on the most appropriate resources.

[0124] Furthermore, based on the acquired real-time resource status data set, the input / output requirements of the task were analyzed, and it was found that some data blocks were allocated to storage nodes far away from the computing nodes, resulting in excessively high cross-node data transmission delays. In this case, the storage locations of the data blocks were readjusted according to the input / output requirements of the task, and frequently accessed data blocks were preferentially stored in storage nodes near the computing nodes, thereby reducing data transmission delays.

[0125] Furthermore, during the continuous execution of the task, it was determined based on the acquired real-time resource status data set that some GPU nodes suddenly experienced resource overload, and the GPU utilization rate rose sharply from 75% to 95%. At this time, according to the scheme recorded in the above embodiment, some tasks can be dynamically migrated to the CPU node. After the adjustment, the GPU utilization rate was restored to 85%, and the CPU utilization rate was increased to 65%, and the system load was effectively balanced.

[0126] The solution of this embodiment can effectively solve the problems of low computing resource utilization, high data transmission delay and unbalanced system load in the prior art. Through intelligent resource scheduling and storage optimization strategies, the computing efficiency and stability of the system are significantly improved, and it can maintain efficient operation in complex task scenarios.

[0127] Embodiment 3

[0128] Figure 3 Schematic diagram of a resource optimization device for a heterogeneous computing cluster according to Embodiment 3 of the present invention. Figure 3 As shown, the device includes: a first real-time resource status data set determination module 310, a determination module 320, a computing resource optimization module 330 and a storage resource optimization module 340.

[0129] The first real-time resource status data set determination module 310 is used to obtain the first usage status of each system resource of each computing node in the distributed system architecture in response to the first processing process of the target task, and obtain the first real-time resource status data set; the system resources include computing resources and storage resources;

[0130] A determination module 320 is used to determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable;

[0131] A computing resource optimization module 330 is used to allocate computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model when it is determined that the computing resource allocation of each computing node in the distributed system architecture is unreasonable;

[0132] The storage resource optimization module 340 is used to adjust the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task when it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable.

[0133] The solution of this embodiment is to obtain the first usage status of each system resource of each computing node in the distributed system architecture in response to the first processing process of the target task through the first real-time resource status data set determination module, and obtain the first real-time resource status data set; the system resources include computing resources and storage resources; the determination module determines whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable; when the computing resource optimization module determines that the allocation of each computing resource of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates the computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and the pre-established task requirement model; when the storage resource optimization module determines that the allocation of each storage resource of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task, so that the system resources of the heterogeneous computing cluster can be dynamically adjusted during the task processing process, thereby improving the execution efficiency of the task and providing a basis for improving the overall performance of the system.

[0134] In an optional implementation of this embodiment, the determination module 320 is specifically configured to input the first real-time resource status data set into a YARN component of the distributed system architecture;

[0135] The YARN component determines whether the allocation of each system resource of each computing node in the current distributed system architecture is reasonable according to the first real-time resource status data set and / or the task requirement of the target task;

[0136] The task requirements include: computing resource requirements and / or storage resource requirements.

[0137] In an optional implementation of this embodiment, the resource optimization device of the heterogeneous computing cluster further includes a task requirement model determination module, which is used to determine the total resource requirement of the target task according to the task load information required by the target task; the task load information includes: the computing power, data transmission rate and storage requirement required to complete the target task;

[0138] Determining a target task requirement model for the target task according to the total resource requirement of the target task;

[0139] Determine a first comprehensive resource requirement of a Map phase and a second comprehensive resource requirement of a Reduce phase in a MapReduce task; the first comprehensive resource requirement includes resource requirements required by all tasks in the distributed system architecture in the Map phase; the second comprehensive resource requirement includes resource requirements required by all tasks in the distributed system architecture in the Reduce phase;

[0140] A comprehensive demand model corresponding to the distributed system architecture is determined according to the first comprehensive resource demand and the second comprehensive resource demand.

[0141] In an optional implementation of this embodiment, the computing resource optimization module 320 is specifically used to determine the adaptability scoring function of each task in each computing resource of the distributed system architecture according to the task requirement model of each task in the distributed system architecture and the first real-time resource status data set;

[0142] Determining the adaptability score of each task in each computing node according to the adaptability score function, and sorting the adaptability scores according to a set order;

[0143] At least one target computing node is determined according to the sorting result, and each of the target computing nodes is used to run the target task.

[0144] In an optional implementation of this embodiment, the storage resource requirement includes: input / output utilization requirement; the input / output utilization requirement includes: read / write speed requirement and data block size requirement;

[0145] Correspondingly, the storage resource optimization module 340 is specifically used for the HDFS component to determine the replication strategy based on the read and write speed requirements and the data block size requirements; and / or, to determine the storage location of each target data block matching the target task based on the read and write speed requirements of the target task, the computing power of each computing node, and the network delay between the target task and each computing node.

[0146] In an optional implementation of this embodiment, the resource optimization device of the heterogeneous computing cluster further includes: an execution strategy adjustment module, which is used to continuously obtain the second usage status of each system resource of each computing node in the distributed system architecture in response to the continuous processing instruction of the target task, and obtain a second real-time resource status data set, as well as the execution progress of the target task;

[0147] Determining a load change function of the target task according to the second real-time resource status data set and the execution progress of the target task;

[0148] The execution strategy of the target task is adjusted according to the load change function.

[0149] In an optional implementation of this embodiment, the resource optimization device for a heterogeneous computing cluster further includes: a feedback module, which is used to store the execution data of the target task in response to the execution completion instruction of the target task, and update the scheduling library in the YARN component based on the execution data;

[0150] The scheduling library stores execution data of different tasks, and the execution data is used to provide a basis for formulating execution strategies for new tasks.

[0151] The resource optimization device for a heterogeneous computing cluster provided in an embodiment of the present invention can execute the resource optimization method for a heterogeneous computing cluster provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] Embodiment 4

[0153] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0154] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0155] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0156] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a resource optimization method for a heterogeneous computing cluster, the method comprising: in response to a first processing process of a target task, obtaining a first usage status of each system resource of each computing node in a distributed system architecture, and obtaining a first real-time resource status data set; the system resources include computing resources and storage resources; determining whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable; when it is determined that the allocation of each computing resource of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates the computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model; when it is determined that the allocation of each storage resource of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task.

[0157] In some embodiments, the resource optimization method for a heterogeneous computing cluster may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the resource optimization method for a heterogeneous computing cluster described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the resource optimization method for a heterogeneous computing cluster in any other appropriate manner (e.g., by means of firmware).

[0158] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0160] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0162] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0163] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0164] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0165] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for optimizing resources of a heterogeneous computing cluster, wherein the heterogeneous computing cluster is deployed in a distributed system architecture, characterized in that: The resource optimization method of the heterogeneous computing cluster includes: In response to a first processing process of the target task, a first usage status of each system resource of each computing node in the distributed system architecture is obtained to obtain a first real-time resource status data set; the system resources include computing resources and storage resources; Determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable; In the case where it is determined that the allocation of computing resources of each computing node in the distributed system architecture is unreasonable, the YARN component of the distributed system architecture allocates computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model; When it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable, the HDFS component of the distributed system architecture adjusts the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task.

2. The resource optimization method for a heterogeneous computing cluster according to claim 1, characterized in that: The determining whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable includes: Inputting the first real-time resource status data set into a YARN component of the distributed system architecture; The YARN component determines whether the allocation of each system resource of each computing node in the current distributed system architecture is reasonable according to the first real-time resource status data set and / or the task requirement of the target task; The task requirements include: computing resource requirements and / or storage resource requirements.

3. The resource optimization method for a heterogeneous computing cluster according to claim 1, characterized in that: The task requirement model is determined by the following steps: Determine the total resource requirement of the target task according to the task load information required by the target task; the task load information includes: the computing power, data transmission rate and storage requirement required to complete the target task; Determining a target task requirement model for the target task according to the total resource requirement of the target task; Determine a first comprehensive resource requirement of a Map phase and a second comprehensive resource requirement of a Reduce phase in a MapReduce task; the first comprehensive resource requirement includes resource requirements required by all tasks in the distributed system architecture in the Map phase; the second comprehensive resource requirement includes resource requirements required by all tasks in the distributed system architecture in the Reduce phase; A comprehensive demand model corresponding to the distributed system architecture is determined according to the first comprehensive resource demand and the second comprehensive resource demand.

4. The resource optimization method for a heterogeneous computing cluster according to claim 3, characterized in that: The YARN component allocates computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model, including: Determining respectively the adaptability scoring function of each task in each computing resource of the distributed system architecture according to the task requirement model of each task in the distributed system architecture and the first real-time resource status data set; Determining the adaptability score of each task in each computing node according to the adaptability score function, and sorting the adaptability scores according to a set order; At least one target computing node is determined according to the sorting result, and each of the target computing nodes is used to run the target task.

5. The resource optimization method for a heterogeneous computing cluster according to claim 1, characterized in that: The storage resource requirements include: input / output utilization requirements; the input / output utilization requirements include: read / write speed requirements and data block size requirements; Accordingly, the HDFS component adjusts the replication strategy and storage location of each data block according to the storage resource requirements in the task requirements of the target task, including: The HDFS component determines the replication strategy according to the read and write speed requirements and the data block size requirements; and / or, The storage locations of each target data block matching the target task are determined according to the read and write speed requirements of the target task, the computing capabilities of each computing node, and the network delay between the target task and each computing node.

6. The method for optimizing resources of a heterogeneous computing cluster according to any one of claims 1 to 5, characterized in that: The resource optimization method of the heterogeneous computing cluster further includes: In response to the continuous processing instruction of the target task, continuously obtain the second usage status of each system resource of each computing node in the distributed system architecture, obtain a second real-time resource status data set, and the execution progress of the target task; Determining a load change function of the target task according to the second real-time resource status data set and the execution progress of the target task; The execution strategy of the target task is adjusted according to the load change function.

7. The resource optimization method for a heterogeneous computing cluster according to claim 1, characterized in that: The resource optimization method of the heterogeneous computing cluster further includes: In response to an execution completion instruction of the target task, storing execution data of the target task, and updating a scheduling library in the YARN component based on the execution data; The scheduling library stores execution data of different tasks, and the execution data is used to provide a basis for formulating execution strategies for new tasks.

8. A resource optimization device for a heterogeneous computing cluster, wherein the heterogeneous computing cluster is deployed in a distributed system architecture, characterized in that: The resource optimization device of the heterogeneous computing cluster includes: A first real-time resource status data set determination module is used to obtain a first usage status of each system resource of each computing node in the distributed system architecture in response to a first processing process of the target task, and obtain a first real-time resource status data set; the system resources include computing resources and storage resources; A determination module, used to determine whether the allocation of each system resource of each computing node in the distributed system architecture is reasonable; A computing resource optimization module, configured to allocate computing resources in the distributed system architecture according to the first real-time resource status data set, the computing resource requirements in the task requirements of the target task, and a pre-established task requirement model when determining that the computing resource allocation of each computing node in the distributed system architecture is unreasonable; The storage resource optimization module is used to adjust the replication strategy and / or storage location of each data block according to the storage resource requirements in the task requirements of the target task when it is determined that the storage resource allocation of each computing node in the distributed system architecture is unreasonable.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the resource optimization method for a heterogeneous computing cluster according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the resource optimization method for a heterogeneous computing cluster according to any one of claims 1 to 7 when executed.

Images