Density peak clustering multi-granularity task decomposition algorithm based on task representation model

Abstract

A density peak clustering multi-granularity task decomposition algorithm based on a task representation model relates to the field of industrial internet, and comprises the following steps: obtaining a computing resource demand vector of each atomic task according to information of each atomic task; performing normalization processing on the computing resource demand vector of each atomic task; calculating a self-calculation resource weight coefficient of the atomic task; calculating correlation parameters among the atomic tasks; calculating distance measurement between the atomic tasks; initializing a cut-off distance; calculating the local density of each atomic task; calculating a minimum center offset distance; obtaining a clustering center; and obtaining a clustering result. According to the method, the representation model of the task is constructed, the resource demand of the task is described, and task clustering is performed by using the weight of the task, so that reasonable resource scheduling and allocation are realized. According to the method, the tasks can be allocated to the most appropriate computing nodes according to the characteristics of the tasks, the system performance is optimized, the task delay is reduced, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial Internet technology, and in particular to a density peak clustering multi-granularity task decomposition algorithm based on a task characterization model. Background Art

[0002] The Industrial Internet is a technology that combines physical systems with information technology to achieve digital interconnection of resources such as equipment, factories, and production lines, and optimizes industrial production processes through data collection, analysis, and feedback. It not only connects industrial equipment to the Internet, but also integrates advanced computing technology, the Internet of Things, cloud computing, edge computing, and big data analysis technology, further improving production efficiency, reducing operating costs, and promoting multi-level comprehensive innovation in industrial models, business models, and technical models.

[0003] Industrial Internet has been widely used in many fields, including automobile manufacturing, smart grid, agriculture, construction, healthcare, etc. These fields have achieved real-time monitoring, predictive maintenance and intelligent optimization through data-driven decision-making, greatly improving the quality of production and services. With the rise of smart manufacturing and smart factories, Industrial Internet has become the core driving force for the digital transformation of manufacturing.

[0004] However, with the expansion of network scale and the increase of task complexity, the traditional centralized cloud computing model is difficult to meet the needs of the Industrial Internet of Things for the transmission and real-time processing of massive data. The characteristics of the Industrial Internet of Things are large data volume, high real-time requirements, and widely distributed nodes. Therefore, unified computing in the cloud center will bring huge network load and affect the response speed and processing efficiency of the system. In order to overcome these challenges, edge computing has been introduced as an important complementary technology to process data on edge devices close to the data source, reduce latency and improve real-time performance.

[0005] The current industrial Internet architecture usually relies on a three-layer architecture of end-edge-cloud: terminal devices (such as sensors and machine equipment) are responsible for data collection and preliminary processing; edge nodes (such as edge servers and edge clusters) are responsible for more complex real-time processing and computing; cloud servers handle complex large-scale tasks and data analysis. Under this architecture, the reasonable allocation of tasks is crucial, especially because nodes at different levels have different resource characteristics and computing capabilities. For example, the computing power of terminal devices is weak and is suitable for processing real-time tasks with low computing power; the computing power of edge nodes is medium, which is suitable for processing tasks with moderate computing power and high timeliness requirements; cloud servers have strong computing and storage capabilities and can handle complex batch tasks.

[0006] Due to the existence of these heterogeneous resources, how to reasonably schedule resources and decompose tasks according to the characteristics of tasks (such as computing volume, real-time requirements, data dependencies, etc.) has become a key challenge in the current industrial Internet field. The resource requirements of tasks vary significantly, which is manifested in different requirements for computing resources such as CPU, GPU, memory, and storage. For example, some tasks may require a large amount of CPU resources, while other tasks may rely on GPUs for fast computing processing. Therefore, in order to maximize the performance and resource utilization of the system, tasks must be analyzed and decomposed at a fine-grained level to ensure that different tasks are reasonably allocated on appropriate nodes.

[0007] When clustering tasks, existing algorithms usually lack a mechanism to fully characterize and measure tasks, and fail to fully consider the multi-dimensional resource requirements of tasks and their complex interrelationships. This results in the fact that the task clustering results may not accurately reflect the characteristics of the actual tasks, which in turn affects the accuracy and efficiency of resource scheduling, especially in complex industrial Internet scenarios.

[0008] In the industrial Internet environment, tasks usually have complex correlations and resource heterogeneity. However, existing algorithms fail to effectively consider the interdependencies between these tasks when performing task decomposition, resulting in inefficient task scheduling. For example, when multiple tasks have data dependencies or shared resources, unreasonable task execution order will seriously affect the execution efficiency and response time of the entire system. In addition, traditional task decomposition algorithms cannot fully consider the heterogeneity of computing node resources and the special needs of tasks, resulting in unreasonable resource scheduling, which further restricts overall production efficiency.

[0009] Although clustering algorithms are important tools for optimizing resource allocation, some existing clustering algorithms perform poorly when faced with high-dimensional data and complex task relationships that are unique to the Industrial Internet. For example, the K-means algorithm requires the number of clusters to be specified in advance and is sensitive to noise and outliers, which can easily lead to deviations in clustering results. The DBSCAN algorithm may encounter difficulties when processing data with large density variations. These limitations make it difficult for them to accurately reflect the complex resource requirements of tasks in task decomposition, thereby affecting the degree of intelligence in scheduling. Therefore, the existing task clustering methods and scheduling algorithms cannot meet the efficient scheduling requirements of actual systems when faced with high heterogeneity and complex task requirements in Industrial Internet scenarios, and are in urgent need of improvement. Summary of the invention

[0010] In view of the characteristics of the current industrial Internet with a wide variety of tasks and different requirements, and in order to effectively solve the performance bottleneck problem in task scheduling, the present invention provides a density peak clustering multi-granularity task decomposition algorithm based on a task characterization model. The present invention proposes a task characterization model for tasks in the industrial Internet to comprehensively characterize and measure tasks, and proposes a density peak clustering multi-granularity task decomposition algorithm based on the task characterization model based on the task characterization model to complete task clustering decomposition. Among them, the industrial Internet task characterization model is mainly based on the properties and correlations of the atomic tasks themselves, and characterizes the entire properties of the task, such as: resource requirements (CPU, GPU, memory, external storage), data exchange volume, bandwidth requirements, data dependency, etc.; and the density peak clustering multi-granularity task decomposition algorithm based on the task characterization model will reasonably decompose the task into multi-granularity sub-tasks according to the measurement of the characterization model, which provides a reasonable choice for the task deployment scheduling of the industrial Internet system to allocate appropriate sub-tasks to appropriate nodes in the mobile edge network, thereby improving the overall performance of the system.

[0011] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0012] The present invention provides a density peak clustering multi-granularity task decomposition algorithm based on a task representation model, which mainly includes the following steps:

[0013] Step S1: Construct an industrial Internet computing task representation model;

[0014] Step S1.1: Based on the information of each atomic task, obtain the computing resource requirement vector of each atomic task;

[0015] Step S1.2: normalize the computing resource requirement vector of each atomic task;

[0016] Step S1.3: Calculate the computing resource weight coefficient of the atomic task itself;

[0017] Step S1.4: Calculate the correlation parameters between atomic tasks;

[0018] Step S2: Perform density peak clustering multi-granularity task decomposition based on the constructed industrial Internet computing task representation model;

[0019] Step S2.1: Calculate the distance metric between atomic tasks;

[0020] Step S2.2: Initialize the cutoff distance;

[0021] Step S2.3: Calculate the local density of each atomic task;

[0022] Step S2.4: Calculate the minimum center offset distance;

[0023] Step S2.5: Obtain cluster centers;

[0024] Step S2.6: Obtain clustering results.

[0025] Furthermore, in step S1.1, for each atomic task AT i , the specific calculation formula of its computing resource requirement vector is:

[0026]

[0027] Among them, CPU i Indicates the CPU demand vector, GPU i Represents the GPU demand vector, Mem i Represents the memory requirement vector, Stor i Represents the external memory requirement vector.

[0028] Furthermore, in step S1.2, the Min-max method is used for normalization processing. After the data x is centered according to the minimum value, it is scaled according to the range, so as to complete the dimensionless transformation of the data and converge to a uniform specification to form a normalized weight:

[0029]

[0030] Among them, W i,x Represents atomic task AT i Normalized weight of data x; x i Represents atomic task AT i The resource demand of a task; max(x) represents the maximum value of data x; min(x) represents the minimum value of data x; the denominator in formula (1), max(x)-min(x), represents the average demand of all tasks on CPU, GPU, memory and external storage.

[0031] Furthermore, in step S1.3, the atomic task itself calculates the resource weight W i The specific calculation formula is:

[0032] W i =λ cpu W i,cpu +λ gpu W i,gpu +λ mem W i,mem +λ Stor W i,stor (2)

[0033] Among them, W i,cpu Represents atomic task AT i The CPU demand weight, W i,gpu Represents atomic task ATi The GPU demand weight, W i,mem Represents atomic task AT i The memory requirement weight, W i,stor Represents atomic task AT i The external memory requirement weight, λ cpu represents the CPU demand adjustment parameter, λ gpu represents the GPU demand adjustment parameter, λ mem represents the memory requirement adjustment parameter, λ stor Indicates the external memory demand adjustment parameter.

[0034] Furthermore, in step S1.4, according to the dependencies and connection factors between the atomic tasks, for any two atomic tasks AT i ,AT j Calculate the correlation coefficient ATR between them ij , the specific calculation formula is:

[0035]

[0036] in, Represents atomic task AT i and atomic tasks AT j Special needs between SD represents the adjustment parameter of the normalized interaction data volume, SD ij Represents atomic task AT i and atomic tasks AT j The normalized interaction data volume between DD The adjustment parameter representing the normalized data dependency, DD ij Represents atomic task AT i and atomic task AT j The normalized data dependency between BD The adjustment parameter representing the normalized bandwidth requirement, BD ij Represents atomic task AT i and atomic tasks AT j The normalized bandwidth requirement between .

[0037] Furthermore, in step S2.1, the specific calculation formula for the distance measurement between the atomic tasks is:

[0038]

[0039] Among them, d ij Represents any two atomic tasks AT i ,AT j The distance metric between i Represents atomic task AT iThe weight of its own computing resources, W j Represents atomic task AT j Self-computing resource weight, ATR ij Represents any two atomic tasks AT i ,AT j The correlation coefficient between .

[0040] Furthermore, in step S2.2, a most suitable cutoff distance is found by minimizing the Gini coefficient; the specific calculation formula of the Gini coefficient G is:

[0041]

[0042] Among them, ρ i Represents atomic task AT i The local density of i Represents atomic task AT i The center offset distance; N represents the total number of data points, that is, the total number of atomic tasks; S represents the sum of the product of local density and the center offset distance, which is used to normalize the contribution value of the atomic task and facilitate the calculation of the Gini coefficient. The specific calculation formula is:

[0043]

[0044] Furthermore, in step S2.3, the specific calculation formula for the local density of each atomic task is:

[0045]

[0046] Among them, d ij Represents any two atomic tasks AT i ,AT j The distance metric between c It represents the cutoff distance, which means that the atomic tasks within a certain range contribute more to the density, and the atomic tasks outside this range have less impact on the density.

[0047] Furthermore, in step S2.4, the specific calculation formula of the minimum center offset distance is as follows:

[0048]

[0049] Among them, d ij Represents any two atomic tasks AT i ,AT j The distance measure between i Represents atomic task AT i The local density, ρ j Represents atomic task AT j Local density.

[0050] Furthermore, in step S2.5 and step S2.6, the cluster center is selected according to the obtained optimal local density and the minimum center offset distance; all atomic tasks are allocated according to the selected cluster center; each atomic task is added to the closest center cluster according to the distance between it and each cluster center, forming several clusters and obtaining a multi-granularity task decomposition solution.

[0051] The beneficial effects of the present invention are:

[0052] The present invention comprehensively characterizes and measures various computing tasks in the industrial Internet, considers the multi-dimensional resource requirements of tasks and the relationships between them, and constructs an industrial Internet computing task characterization model; based on the constructed industrial Internet computing task characterization model, density peak clustering multi-granularity task decomposition is performed: based on the task characterization model, a series of related atomic tasks are combined into composite subtasks through an improved algorithm to form a multi-granularity task cluster in order to optimize scheduling and resource allocation. Compared with the prior art, the present invention has the following advantages:

[0053] Based on the Density Peaks Clustering (DPC) algorithm and the computing task characterization model, the present invention proposes a density peak clustering multi-granularity task decomposition algorithm based on the task characterization model for the resource allocation problem of computing tasks in the industrial Internet. The industrial Internet computing task characterization model constructed by the present invention comprehensively considers the demand for computing resources of different tasks, the correlation between tasks, and the characteristics of task allocation in the industrial Internet environment, aiming to achieve efficient allocation of heterogeneous computing resources. Through a multi-granularity clustering method, the present invention can decompose tasks into different sub-task groups according to the resource requirements and correlation of tasks, thereby optimizing resource allocation and improving task execution efficiency. The effectiveness of the present invention has been verified by simulation experiments. The density peak clustering multi-granularity task decomposition algorithm based on the task characterization model provided by the present invention has better performance than other similar algorithms. The present invention is particularly suitable for scenarios with diversified tasks and heterogeneous resources in the industrial Internet of Things, and can effectively solve the performance bottleneck problem in task scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A flow chart of a density peak clustering multi-granularity task decomposition algorithm based on a task representation model provided by the present invention.

[0055] Figure 2 It is a relationship style diagram of atomic tasks.

[0056] Figure 3 It is the algorithm decision diagram.

[0057] Figure 4 This is the algorithm result diagram with a granularity of 2.

[0058] Figure 5 This is the algorithm result diagram with a granularity of 3.

[0059] Figure 6 This is the algorithm result diagram with a granularity of 4.

[0060] Figure 7 It is the algorithm evaluation index result data (granularity is 2).

[0061] Figure 8 It is the algorithm evaluation index result data (granularity is 3).

[0062] Fig. 9 It is the algorithm evaluation index result data (granularity is 4). DETAILED DESCRIPTION

[0063] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the present invention provides a density peak clustering multi-granularity task decomposition algorithm based on the task representation model, and its specific implementation process is as follows:

[0065] Step 1: Construction of the industrial Internet computing task representation model;

[0066] Tasks in the Industrial Internet are usually diverse and heterogeneous, and different tasks have significantly different requirements for computing resources, storage resources, communication bandwidth, etc. The distributed architecture of the Industrial Internet usually consists of three layers: end, edge, and cloud, and the computing power between the layers varies greatly. The cloud side has powerful computing and storage capabilities, which is suitable for large-scale data analysis and complex computing tasks, but has high latency; the edge side is usually an edge server or cluster deployed inside the factory, with medium computing power and storage capacity, suitable for processing tasks with high timeliness requirements; the end side is usually various types of terminal live sensors, with less computing power, and can only perform data collection or simple processing.

[0067] For various tasks in the Industrial Internet, the representation model is the basis of the entire clustering process. It needs to be comprehensively characterized and measured, considering the multi-dimensional resource requirements of the tasks and their interrelationships, so as to optimize the results of task clustering.

[0068] An atomic task is a basic task unit in an application that cannot be further subdivided in the context of the Industrial Internet. It represents a minimum executable operation or computing unit in the system, usually with clear computing requirements and resource consumption. Atomic tasks can be seen as the basic components of complex computing tasks. By combining these atomic tasks together, more complex tasks and applications can be built.

[0069] 1) Based on the information of each atomic task, obtain the computing resource requirement vector of each atomic task;

[0070] In the Industrial Internet, the computing resource requirements of different atomic tasks are significantly different. In order to effectively characterize and support subsequent clustering and effective task allocation and scheduling, it is necessary to characterize the task's requirements for various computing resources in detail. Specifically, computing resource requirements can be analyzed and measured from four dimensions: CPU requirements, GPU requirements, memory requirements, and external storage requirements.

[0071] For each atomic task AT i , its computational resource requirements can be expressed as a vector:

[0072]

[0073] Among them, CPU i Indicates the CPU demand vector, GPU i Represents the GPU demand vector, Mem i Represents the memory requirement vector, Stor i Represents the external memory requirement vector.

[0074] In this vector, each component corresponds to the task's requirements for CPU, GPU, memory, and external storage.

[0075] 2) Normalize the computing resource requirement vector of each atomic task;

[0076] Since different tasks have different requirements for computing resources, these computing resources need to be normalized. Normalization can effectively map different types of computing resource requirements to the same scale range, avoiding misjudgment caused by differences in the magnitude of the original data.

[0077] In the present invention, the Min-max method is specifically used for normalization processing, that is, the data x is centered according to the minimum value, and then scaled according to the range (maximum value-minimum value), so as to complete the dimensionless transformation of the data and converge to a unified specification to form a normalized weight:

[0078]

[0079] Among them, W i,x Represents atomic task AT i Normalized weight of data x; x i Represents atomic task AT iThe resource requirement of a task is: max(x) represents the maximum value of data x; min(x) represents the minimum value of data x; the denominator in formula (1), max(x)-min(x), represents the average requirement of all tasks on CPU, GPU, memory and external storage. By normalizing the rows, the differences in the order of magnitude and dimension of computing resource requirements of different tasks can be eliminated.

[0080] 3) Calculate the computing resource weight coefficient of the atomic task itself;

[0081] The present invention also introduces adjustment parameters to control the influence of various computing resources in the total weight, so that the model has better adaptability. The adjustment parameters can be dynamically adjusted according to the needs of the actual application scenario to ensure the reasonable allocation of computing resources in different environments. For example, in some scenarios, GPU resources may be more scarce than CPU resources, so the GPU demand adjustment parameter λ can be appropriately increased. gpu The value of is used to give priority to GPU-intensive tasks. The detailed information is shown in Table 1.

[0082] Table 1 Detailed information of atomic task computing resources

[0083]

[0084] In summary, the computing resource weight W of each atomic task can be obtained for each atomic task. i , thereby ensuring that multi-dimensional resources can be examined and measured, and ensuring that factors such as their own needs and resource heterogeneity can be considered simultaneously during clustering.

[0085] Among them, the atomic task AT i The weight of its own computing resources W i The specific calculation formula is as follows:

[0086] W i =λ cpu W i,cpu +λ gpu W i,gpu +λ mem W i,mem +λ Stor W i,stor (2)

[0087] Among them, W i,cpu Represents atomic task AT i The CPU demand weight, W i,gpu Represents atomic task AT i The GPU demand weight, W i,mem Represents atomic task AT i The memory requirement weight, W i,stor Represents atomic task AT iThe external memory requirement weight, λ cpu represents the CPU demand adjustment parameter, λ gpu represents the GPU demand adjustment parameter, λ mem represents the memory requirement adjustment parameter, λ stor Indicates the external memory demand adjustment parameter.

[0088] By characterizing the computing resource requirements of each atomic task in various dimensions in detail, a weight related to its computing resource requirements can be established for each atomic task element, ensuring that the differences in computing resource requirements of atomic tasks can be fully considered during clustering.

[0089] 4) Calculate the correlation parameters between atomic tasks;

[0090] When building a representation model, it is also necessary to consider the relationship measurement between atomic tasks. The correlation between tasks is reflected in multiple dimensions, especially in data exchange, bandwidth requirements, data dependencies, and communication delays. Reasonable consideration of these correlations helps optimize the clustering results of atomic tasks, avoid excessive bottlenecks and unbalanced resource allocation during communication and data transmission, and thus improve the overall performance of the system.

[0091] In order to quantify and characterize the relationship between atomic tasks, the following parameters are used to describe the dependencies and connections between atomic tasks: interactive data volume, data dependency, bandwidth requirements, and special requirements. The detailed parameters are shown in Table 2.

[0092] Table 2. Correlation parameters between atomic tasks

[0093]

[0094] Thus, the correlation coefficient ATR between atomic tasks can be calculated ij , the specific calculation formula is as follows:

[0095]

[0096] in, Represents atomic task AT i and atomic tasks AT j Special needs between SD represents the adjustment parameter of the normalized interaction data volume, SD ij Represents atomic task AT i and atomic tasks AT j The normalized interaction data volume between DD The adjustment parameter representing the normalized data dependency, DD ij Represents atomic task AT i and atomic tasks AT jThe normalized data dependency between BD The adjustment parameter representing the normalized bandwidth requirement, BD ij Represents atomic task AT i and atomic tasks AT j The normalized bandwidth requirement between .

[0097] According to the dependencies and connection factors between atomic tasks, for any two atomic tasks AT i ,AT j Calculate the correlation coefficient ATR between them ij , providing a basis for the subsequent calculation of the distance and local density between atomic task elements. These correlation coefficients reflect the closeness between atomic task elements and are the core basis for the subsequent calculation of atomic task distance measurement.

[0098] The computing resource requirement vector of each atomic task obtained through the above calculation indicates the computing resource requirements of each dimension of the atomic task. At the same time, according to the nature of each atomic task, the associated parameters between atomic tasks are obtained, and according to the scenario requirements, the necessary adjustment parameters are set to complete the construction of the industrial Internet computing task representation model.

[0099] Step 2: Multi-granularity task decomposition based on density peak clustering of industrial Internet computing task representation model;

[0100] When decomposing tasks in the industrial Internet, it is necessary to consider the functional composition, resource requirements, and relationships of each atomic task, which is relatively complex. Therefore, an effective method is needed to classify and group tasks, and to combine a series of related atomic tasks into composite subtasks to form task clusters in order to optimize scheduling and resource allocation. Clustering algorithms can help divide tasks into different clusters and find the connections and trends between atomic tasks in the absence of supervisory information.

[0101] Clustering algorithm is an unsupervised learning method used to group data into different clusters, so that the data in the same cluster is more similar, while the data in different clusters is less similar. Density Peaks Clustering (DPC) is a density-based clustering algorithm. Its core idea is to determine the cluster center based on the distance measurement between the local density of the data and the points with higher density. Its advantage is that density peak clustering can determine the number of clusters based on the density distribution, can better deal with noise and outliers in the data, can effectively process data clusters with different shapes and densities, and discover the natural structure contained in the data. Its advantages as above are just suitable for the complexity of atomic tasks in the industrial Internet and the heterogeneity of processing resources, which is significantly improved compared with other traditional clustering algorithms. Therefore, a density peak clustering multi-granularity task decomposition algorithm based on the task representation model is proposed.

[0102] 1) Calculate the distance metric between atomic tasks;

[0103] In the density peak clustering algorithm, the distance metric d ij It is a crucial link, which is used to measure the similarity between various data points (referring to atomic tasks in this invention). In task decomposition and resource scheduling, a reasonable distance metric can effectively reflect the mutual relationship between tasks and the similarities and differences in resource requirements, thereby helping to cluster related tasks into the same cluster. A good distance metric not only needs to be able to reflect the differences in resource requirements between atomic tasks, but also needs to be able to capture the dependencies between tasks. Therefore, in this invention, a distance metric based on representation is proposed, and its specific calculation formula is as follows:

[0104]

[0105] Among them, d ij Represents any two atomic tasks AT i ,AT j The distance metric between i Represents atomic task AT i The weight of its own computing resources, W j Represents atomic task AT j Self-computing resource weight, ATR ij Represents any two atomic tasks AT i ,AT j The correlation coefficient between .

[0106] The distance metric can consider both the properties of the atomic task itself and the related properties between the atomic tasks. This ensures that tasks with large resource requirements have a greater impact on the distance calculation. At the same time, the stronger the correlation between tasks, the smaller the distance, which more accurately reflects the relationship between tasks.

[0107] 2) Initialize the cutoff distance;

[0108] Cutoff distance d c It is also critical that it determines which atomic tasks will be considered as closely related neighbors, thus affecting the calculation of local density. In the traditional density peak clustering algorithm, the cutoff distance d c It is usually selected manually or set based on experience. This subjectivity may lead to inaccurate or unstable clustering results. Therefore, the present invention introduces the Gini Coefficient to optimize the truncation distance selection. The specific calculation formula of the Gini Coefficient G is as follows:

[0109]

[0110] Among them, ρ i Represents atomic task AT i The local density of i Represents atomic task AT i The center offset distance; N represents the total number of data points, that is, the total number of atomic tasks; S represents the sum of the product of local density and the center offset distance, which is used to normalize the contribution value of the atomic task and facilitate the calculation of the Gini coefficient. The specific calculation formula is as follows:

[0111]

[0112] By minimizing the Gini coefficient, we can find the most suitable cutoff distance so that the local density distribution of atomic tasks is balanced, thus obtaining a reasonable clustering result.

[0113] 3) Calculate the local density of each atomic task;

[0114] Local density ρ i It is also a core parameter, which is used to measure the data distribution density of a certain atomic task in its neighborhood. In the density peak clustering algorithm, the local density reflects the density of other atomic tasks around the atomic task, and determines whether the atomic task is likely to become a cluster center. The introduction of local density has two important functions: first, it can help the algorithm find areas with higher local density, so that the atomic tasks in these areas are more likely to be selected as cluster centers; second, it can also enhance the robustness of the algorithm to noise, because noise points in sparse areas usually have lower local density, thus avoiding being mistaken for cluster centers. Local density ρ i The specific calculation formula is as follows:

[0115]

[0116] Among them, d ij Represents any two atomic tasks ATi ,AT j The distance metric between c It represents the cutoff distance, which means that the atomic tasks within a certain range contribute more to the density, and the atomic tasks outside this range have less impact on the density. In this way, the density peak clustering algorithm can identify the atomic tasks with high density and far away from other high-density atomic tasks as cluster centers.

[0117] 4) Calculate the minimum center offset distance;

[0118] With the local density ρ i A related parameter is the center offset distance, which is used to determine the distance of each atomic task relative to atomic tasks with higher local density. The center offset distance can help distinguish those atomic tasks with higher local density and farther away from other high-density atomic tasks, making it easier to identify cluster centers. Specifically, the minimum center offset distance is defined as: for any atomic task AT i , its minimum center offset distance δ i Refers to the atomic task AT i With all local densities greater than the atomic task AT i The minimum distance between atomic tasks is calculated as follows:

[0119]

[0120] Among them, d ij Represents any two atomic tasks AT i ,AT j The distance measure between i Represents atomic task AT i The local density, ρ j Represents atomic task AT j Local density.

[0121] 5) Obtain cluster centers;

[0122] The cluster center is selected according to the optimal local density and the minimum center offset distance obtained above.

[0123] 6) Obtain clustering results;

[0124] All atomic tasks are assigned according to the selected cluster centers. Each atomic task is added to the closest center cluster according to the distance between it and each cluster center, so that several clusters can be formed, and finally a multi-granular task decomposition solution is obtained.

[0125] The present invention provides a density peak clustering multi-granularity task decomposition algorithm (MDPC-TM) based on a task characterization model. In order to verify the effectiveness of the MDPC-TM algorithm, the K-Means algorithm and the IDPC-Dcom algorithm are used as comparison schemes, and the full decomposition (Full Dcom) method is used as a baseline reference. Clustering experimental operations are performed on all atomic tasks participating in the clustering decomposition, and the clustering decomposition results are compared.

[0126] In this embodiment, the computer hardware of the test environment is 12th Gen Intel Core i7-12700H (2.30GHz), and the operating system version based on it is Windows 1123H2. The algorithm is implemented based on Python 3.9, and the version of scikit-learn that it relies on is 1.0.2.

[0127] 1. Experimental methods and evaluation indicators

[0128] The experiment will evaluate the performance of the algorithm from the following evaluation indicators: the amount of inter-cluster interaction data, the amount of inter-cluster bandwidth required, and the inter-cluster data dependency. These evaluation indicators can effectively evaluate the effect of each sub-task group that forms a cluster after the task clustering decomposition.

[0129] 1) Inter-cluster interaction data volume

[0130] Meaning: After the algorithm is executed, a series of clusters can be obtained. Each result cluster after task clustering is regarded as a subtask set as a whole. The amount of inter-cluster interaction data refers to the sum of data that needs to be exchanged between each cluster, which is used to characterize the amount of information that needs to be transmitted between clusters after task classification.

[0131] Evaluation significance: When a clustering algorithm is good, the amount of data interaction between clusters is small, because tasks that require a large amount of data transmission between atomic tasks can be clustered into the same cluster and deployed on the same server or the same cluster, converting a large amount of network IO into information transmission within the server or cluster, improving service performance while reducing the consumption of network resources and reducing the negative impact of network fluctuations on services.

[0132] 2) Inter-cluster bandwidth requirement

[0133] Meaning: Inter-cluster bandwidth demand measures the total network bandwidth required between clustered task clusters. For each pair of clusters, if there is a need for data exchange between their tasks, a certain amount of bandwidth is required to transmit data. The smaller the bandwidth demand, the better the algorithm can concentrate tasks with high bandwidth requirements in the same cluster in task division, thereby reducing the bandwidth occupied between clusters.

[0134] Evaluation significance: When the clustering algorithm works well, the bandwidth demand between clusters is low, which means that after the task division, the tasks with high bandwidth demand are effectively concentrated, reducing the bandwidth demand between clusters, and at the same time reducing the possibility of network bottlenecks caused by bandwidth demand, and reducing the demand for bandwidth resources. On the contrary, a large bandwidth demand means that there is more communication between clusters and a high demand for network bandwidth resources, which may cause problems such as network transmission delay or insufficient bandwidth, which is not conducive to the execution of real-time tasks, and will increase the system load and lead to network resource tension.

[0135] 3) Inter-cluster data dependency

[0136] Meaning: Inter-cluster data dependency is used to measure the data dependency between tasks in each cluster after task division. Tasks with high data dependency usually rely on the same input or upstream data stream. If these tasks are assigned to different clusters, multiple unnecessary data copies may be generated to meet the needs of inter-cluster data exchange. Such data dispersion not only increases the communication overhead between clusters, but also creates an additional burden on network resources due to transmission requirements. At the same time, the process of generating data copies will also consume additional CPU computing resources and network IO, thereby reducing the overall system efficiency. Therefore, inter-cluster data dependency can be used as an important indicator to evaluate the rationality of task clustering.

[0137] Evaluation significance: Inter-cluster data dependency reflects the data dependency status between clusters after task clustering. Ideally, the clustering algorithm should try to aggregate tasks with high data dependency into the same cluster, thereby reducing the transmission requirements between clusters and improving system performance and resource utilization. If the inter-cluster data dependency is low, it means that the clustering result effectively reduces the data coupling across clusters and the task allocation is reasonable; high data dependency means that the data traffic across the network will increase, and additional data copies may increase the consumption of resources such as CPU, IO, and memory, resulting in resource fragmentation and low resource utilization.

[0138] The experimental data set is the same as the control algorithm, which is a set of computing tasks containing 9 subtasks. The data set contains the CPU demand, GPU demand, memory demand, and external storage demand of each task. At the same time, the data set contains the amount of interaction data, data dependency, and bandwidth demand between atomic tasks. The specific number of resource nodes is set to 3, so the cluster granularity of clustering is selected as 2, 3, and 4. In order to balance various resources, the adjustment parameters in the experiment are selected in average mode, and no offset is made for specific resources and associations. The specific experimental steps are as follows:

[0139] Step 1: Based on the relationship between each atomic task, export the association relationship style diagram of all atomic tasks in the application, such as Figure 2 shown.

[0140] Step 2: The appropriate cutoff distance is derived through the adaptive constraint distance dc based on the Gini parameter optimized by the algorithm, and a decision graph is formed to obtain the constraint density and the minimum relative distance.

[0141] Step 3: Generate multi-granularity results (the number of clusters is 2, 3, and 4), obtain the cluster centers and clustering results, and compare the results of other algorithms at this granularity according to the selected evaluation indicators to verify the algorithm.

[0142] (II) Experimental results

[0143] After running the algorithm, you can get the decision diagram of the clustering algorithm and the decomposition results of different granularities. The decision diagram is used to help identify the cluster center in algorithms such as density peak clustering. It visualizes the distribution of data points and clustering characteristics by showing the relationship between the local density and distance of data points. Figure 3 As shown, it can be seen from the decision diagram that point 0, point 2, point 4, and point 5 are obviously far away from other data points, indicating that they can be used as cluster centers. When selecting cluster centers, the present invention selects different numbers of cluster centers according to different granularities, thereby realizing multi-granularity cluster decomposition.

[0144] The clustering results with granularity of 2, 3, and 4 are shown below: Figure 4 , Figure 5 , Figure 6 shown.

[0145] According to the evaluation indicators, the amount of inter-cluster interaction data, inter-cluster bandwidth requirements, and inter-cluster data dependency parameters of the MDPC-TM algorithm, K-Means algorithm, and IDPC-Dcom algorithm at different granularities are calculated respectively. For the convenience of intuitive comparison, all results are normalized based on the baseline algorithm Full Dcom.

[0146] 1) Scenario 1: The clustering granularity is 2;

[0147] The evaluation indexes of each algorithm are given in Table 3 when the granularity is 2. In order to more intuitively show the performance of each algorithm under different evaluation indicators, a bar chart is drawn as follows: Figure 7 shown.

[0148] Table 3 Algorithm evaluation index result data (granularity is 2)

[0149] algorithm The amount of data exchanged between clusters Inter-cluster data dependencies Inter-cluster bandwidth requirement MDPC-TM 0.192533 0.088235 0.168835 IDPC-Dcom 0.216666 0.071786 0.206186 K-Means 0.27416 0.098124 0.228133 FullDcom 1 1 1

[0150] From Table 3 and Figure 7As can be seen from the figure, compared with Full Dcom, the three algorithms can optimize the amount of data exchanged between clusters, the data dependency between clusters, and the bandwidth requirements between clusters. The amount of data exchanged between clusters of MDPC-TM at a granularity of 2 is about 0.19, which is significantly smaller than that of IDPC-Dcom and K-Means. The smaller amount of data exchanged between clusters indicates that MDPC-TM can effectively aggregate tasks that require a large amount of data exchange in the same cluster, thereby reducing the amount of data exchange between different clusters.

[0151] In terms of inter-cluster data dependency, the value of MDPC-TM is close to that of IDPC-Dcom, slightly higher than that of IDPC-Dcom, but significantly better than K-Means. The lower inter-cluster data dependency indicates that MDPC-TM better allocates tasks with strong data dependency to the same cluster when dividing tasks, thereby effectively reducing the coupling of data across clusters, which can help reduce data traffic and avoid adding additional data copies that occupy various resources.

[0152] In the comparison of bandwidth requirements between clusters, MDPC-TM performs best, indicating that low bandwidth requirements indicate that MDPC-TM can reduce bandwidth usage between different clusters, which is beneficial to optimizing the overall performance of the system and saving network resources.

[0153] 2) Scenario 2: The clustering granularity is 3;

[0154] When the granularity of each algorithm evaluation index is 3, the result data is shown in Table 4, and the comparison chart is as follows Figure 8 shown.

[0155] Table 4 Algorithm evaluation index result data (granularity is 3)

[0156] algorithm The amount of data exchanged between clusters Inter-cluster data dependencies Inter-cluster bandwidth requirement MDPC-TM 0.363636 0.264706 0.421127 IDPC-Dcom 0.383333 0.294416 0.486835 K-Means 0.473156 0.305354 0.548932 FullDcom 1 1 1

[0157] From Table 4 and Figure 8 As can be seen from the figure, as the granularity increases to 3, the amount of data exchanged between clusters of MDPC-TM increases compared to the granularity of 2, but it is still better than IDPC-Dcom and K-Means. This result further verifies that MDPC-TM can maintain a relatively low amount of data exchange when the task allocation granularity is increased, and maintains its advantage in controlling the amount of data exchanged between clusters.

[0158] In terms of inter-cluster data dependency, MDPC-TM performs better than IDPC-Dcom and K-Means. This shows that as the granularity increases, MDPC-TM can reasonably divide the dependencies between tasks and reduce cross-cluster dependencies by multi-dimensionally processing the properties of atomic tasks and the relationships between atomic tasks in the process of characterizing model construction.

[0159] For bandwidth demand, MDPC-TM performs best, which shows its good task clustering ability and can reduce bandwidth usage among different task clusters.

[0160] 3) Scenario 3: The clustering granularity is 4;

[0161] When the granularity of each algorithm evaluation index is 4, the result data are shown in Table 5 and the comparison chart is as follows: Fig. 9 shown.

[0162] Table 5 Algorithm evaluation index result data (granularity is 4)

[0163] algorithm The amount of data exchanged between clusters Inter-cluster data dependencies Inter-cluster bandwidth requirement MDPC-TM 0.533427 0.395214 0.695463 IDPC-Dcom 0.633333 0.413625 0.747621 K-Means 0.785108 0.531288 0.781027 FullDcom 1 1 1

[0164] In the case of a particle size of 4, it can be seen from Table 6 and Fig. 9 It is observed that MDPC-TM is significantly better than other compared algorithms in terms of the amount of inter-cluster exchanged data, inter-cluster data dependency, and inter-cluster bandwidth requirements.

[0165] In summary, the rationality and superiority of the density peak clustering multi-granularity task decomposition algorithm based on the task characterization model of the present invention are verified. Especially in the case of higher granularity, MDPC-TM can still better control the data exchange, data dependency and bandwidth requirements between clusters, and has excellent performance.

[0166] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-granularity task decomposition algorithm based on density peak clustering based on task representation model, characterized by: The following steps are involved: Step S1: Construct an industrial Internet computing task representation model; Step S1.1: Based on the information of each atomic task, obtain the computing resource requirement vector of each atomic task; Step S1.2: normalize the computing resource requirement vector of each atomic task; Step S1.3: Calculate the computing resource weight coefficient of the atomic task itself; Step S1.4: Calculate the correlation parameters between atomic tasks; Step S2: Perform density peak clustering multi-granularity task decomposition based on the constructed industrial Internet computing task representation model; Step S2.1: Calculate the distance metric between atomic tasks; Step S2.2: Initialize the cutoff distance; Step S2.3: Calculate the local density of each atomic task; Step S2.4: Calculate the minimum center offset distance; Step S2.5: Obtain cluster centers; Step S2.6: Obtain clustering results.

2. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S1.1, for each atomic task AT i , the specific calculation formula of its computing resource requirement vector is: Among them, CPU i Indicates the CPU demand vector, GPU i Represents the GPU demand vector, Mem i Represents the memory requirement vector, Stor i Represents the external memory requirement vector.

3. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S1.2, the Min-max method is used for normalization. After the data x is centered according to the minimum value, it is scaled according to the range, so as to complete the dimensionless transformation of the data and converge to a uniform specification to form a normalized weight: Among them, W i,x Represents atomic task AT i Normalized weight of data x; x i Represents atomic task AT i The resource demand of a task; max(x) represents the maximum value of data x; min(x) represents the minimum value of data x; the denominator in formula (1), max(x)-min(x), represents the average demand of all tasks on CPU, GPU, memory and external storage.

4. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S1.3, the atomic task itself calculates the resource weight W i The specific calculation formula is: W i =λ cpu W i,cpu +λ gpu W i,gpu +λ mem W i,mem +λ Stor W i,stor (2) Among them, W i,cpu Represents atomic task AT i The CPU demand weight, W i,gpu Represents atomic task AT i The GPU demand weight, W i,mem Represents atomic task AT i The memory requirement weight, W i,stor Represents atomic task AT i The external memory requirement weight, λ cpu represents the CPU demand adjustment parameter, λ gpu represents the GPU demand adjustment parameter, λ mem represents the memory requirement adjustment parameter, λ stor Indicates the external memory demand adjustment parameter.

5. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S1.4, according to the dependencies and connection factors between atomic tasks, for any two atomic tasks AT i ,AT j Calculate the correlation coefficient ATR between them ij , the specific calculation formula is: in, Represents atomic task AT i and atomic task AT j Special needs between SD represents the adjustment parameter of the normalized interaction data volume, SD ij Represents atomic task AT i and atomic tasks AT j The normalized interaction data volume between DD The adjustment parameter representing the normalized data dependency, DD ij Represents atomic task AT i and atomic tasks AT j The normalized data dependency between BD The adjustment parameter representing the normalized bandwidth requirement, BD ij Represents atomic task AT i and atomic tasks AT j The normalized bandwidth requirement between .

6. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S2.1, the specific calculation formula for the distance measurement between the atomic tasks is: Among them, d ij Represents any two atomic tasks AT i ,AT j The distance metric between i Represents atomic task AT i The weight of its own computing resources, W j Represents atomic task AT j Self-computing resource weight, ATR ij Represents any two atomic tasks AT i ,AT j The correlation coefficient between .

7. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S2.2, a most suitable cutoff distance is found by minimizing the Gini coefficient; the specific calculation formula of the Gini coefficient G is: Among them, ρ i Represents atomic task AT i The local density of i Represents atomic task AT i The center offset distance; N represents the total number of data points, that is, the total number of atomic tasks; S represents the sum of the product of local density and the center offset distance, which is used to normalize the contribution value of the atomic task and facilitate the calculation of the Gini coefficient. The specific calculation formula is:

8. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S2.3, the specific calculation formula for the local density of each atomic task is: Among them, d ij Represents any two atomic tasks AT i ,AT j The distance metric between c It represents the cutoff distance, which means that the atomic tasks within a certain range contribute more to the density, and the atomic tasks outside this range have less impact on the density.

9. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S2.4, the specific calculation formula of the minimum center offset distance is as follows: Among them, d ij Represents any two atomic tasks AT i ,AT j The distance measure between i Represents atomic task AT i The local density, ρ j Represents atomic task AT j Local density.

10. The density peak clustering multi-granularity task decomposition algorithm based on the task representation model according to claim 1 is characterized in that: In step S2.5 and step S2.6, the cluster center is selected according to the obtained optimal local density and the minimum center offset distance; all atomic tasks are allocated according to the selected cluster center; each atomic task is added to the closest center cluster according to the distance between it and each cluster center, forming several clusters and obtaining a multi-granularity task decomposition solution.

Images