Heterogeneous computing power fusion and dynamic optimization distribution method and system

Abstract

The invention discloses a heterogeneous computing power fusion and dynamic optimization allocation method and system, relates to the technical field of computing power allocation, and aims to solve the problem that a receiving task cannot be better fused with heterogeneous computing power. According to the method, real-time data analysis is carried out on the monitoring index of each node, so that more refined computing power resource management can be realized. By analyzing the data, the problems of unbalanced resource use, overhigh node load and the like can be found, and accurate mapping among the data is realized by defining key fields in computing power task allocation data and computing power node connection data. This helps to ensure that tasks are correctly allocated to nodes with appropriate resources and states. And after data mapping is completed, network layer fusion is carried out, which is beneficial for realizing more efficient task execution and resource utilization in computing power allocation.

Description

Technical Field

[0001] The present invention relates to the field of computing power allocation technology, and specifically to a method and system for heterogeneous computing power integration and dynamic optimization allocation. Background Art

[0002] Computing power allocation refers to the process of rationally allocating the processing power required for computing tasks to available computing resources.

[0003] The Chinese patent with publication number CN116862126A discloses a method and device for allocating computing power under the fusion of energy network and computing power network, which mainly uses the green power ratio and remaining computing power of each computing power node in the energy computing power allocation, wherein the computing demand instruction includes the task to be calculated and the computing power requirement data, and determines the current allocable node according to the green power ratio and remaining computing power of each computing power node in the energy computing power allocation, and allocates the computing demand instruction to the allocable node, so that the allocable node performs calculation according to the computing demand instruction. Although the above patent document solves the problem of computing power allocation, the following problems still exist: 1. Failure to make targeted load strategy adjustments based on the specific circumstances of the actual request task, resulting in reduced data load capacity.

[0004] 2. The received tasks are not integrated with the current network layer in a targeted manner, resulting in poor computing power data integration.

[0005] 3. There is no real-time monitoring of the implementation process, which results in the inability to quickly warn of abnormal data during the implementation process. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for heterogeneous computing power fusion and dynamic optimization allocation, and to perform real-time data analysis on the monitoring indicators of each node, which helps to achieve more refined computing power resource management. By analyzing the data, problems such as unbalanced resource use and excessive node load can be found. By clarifying the key fields in the computing power task allocation data and the computing power node connection data, accurate mapping between data is achieved. This helps to ensure that tasks are correctly assigned to nodes with appropriate resources and status. After the data mapping is completed, network layer fusion is performed, which helps to achieve more efficient task execution and resource utilization in computing power allocation, and can solve problems in the prior art.

[0007] To achieve the above object, the present invention provides the following technical solutions: The heterogeneous computing power integration and dynamic optimization allocation method includes the following steps: S1: Confirmation of central computing power: collect the hardware configuration information in the acquisition interface, process the collected hardware configuration information and extract key parameters, perform benchmark tests based on the extracted key parameters, and obtain real-time computing power data after the benchmark tests; S2: Load balancing strategy formulation: load analysis is performed based on real-time computing power data and standard computing power data, and load balancing strategy is formulated based on the load situation. After the load balancing strategy is formulated, the real-time computing power strategy data is obtained; S3: Node task allocation: Analyze the task requirements of the received task requests, and use the real-time computing power strategy data to allocate the analyzed tasks to the nodes. After the node task allocation is completed, the computing power task allocation data is obtained; S4: Node information collaborative processing: Establish a node communication mechanism for each node in the computing power task allocation data, and obtain computing power node connection data after the communication mechanism is established; S5: Processing task network fusion management: Map the computing power task allocation data with the computing power node connection data, and perform network layer fusion after the data mapping is completed. After the network layer fusion, the standard computing power task allocation data is obtained.

[0008] Preferably, in step S1, the hardware configuration information in the acquisition interface is collected, and the collected hardware configuration information is processed and key parameters are extracted, and a benchmark test is performed according to the extracted key parameters, including: The hardware configuration information in the acquisition interface includes processor information, memory information, storage device information, network interface information, computing unit and resource usage information; After the hardware configuration information is collected, the key parameters of the hardware configuration information are extracted; Among them, the key parameters of processor information include the number of cores, number of threads, main frequency, maximum acceleration frequency and cache size; the key parameters of memory information include total capacity, type, speed and timing; the key parameters of storage device information include type, capacity, read and write speed and interface type; the key parameters of network interface information include type, speed, MAC address and IP address; the key parameters of computing unit include model, number of CUDA cores, video memory size, video memory type and computing power; the key parameters of resource usage information include COU usage, memory usage, disk frequency, disk speed and network traffic; The key parameters of the hardware configuration information are extracted and benchmarked; The test results are confirmed according to the benchmark test results, and the confirmed test results are used to generate real-time computing power data.

[0009] Preferably, in step S2, load analysis is performed based on the real-time computing capacity data and the standard computing capacity data, and a load balancing strategy is formulated based on the load condition, including: Retrieving standard computing capacity data from a database, where the standard computing capacity data is a standard expected performance indicator or a statistical average of historical performance data; Compare the real-time computing capability data with the performance indicators with the same attributes in the standard computing capability data, and calculate the difference value of each performance indicator based on the data comparison result; Perform load analysis based on the difference value; The load conditions are divided into normal load, light load and heavy load. When the percentage of the difference value is within the range of 80%-100%, it is a normal load; when the percentage of the difference value is less than 80%, it is a light load; when the percentage of the difference value is greater than 100%, it is a heavy load. A load analysis matrix is ​​constructed based on the analyzed load conditions, and load balancing operation marking is performed based on the constructed load matrix; Formulate load balancing strategies based on the load balancing operations marked in the load analysis matrix. Load balancing strategies include task migration, traffic regulation, and resource reallocation. After the load balancing strategy is formulated, real-time computing capacity strategy data is generated.

[0010] Preferably, performing task demand analysis on the task request received in step S3 and using real-time computing capability strategy data to allocate the analyzed tasks to nodes includes: When the computing network receives a task request from an upper-layer application, it parses the task requirements of the task request; The parsed task requirements include task type, resource requirements, execution time, priority, and data dependencies; The task requirement data is matched and calculated using the task matching calculation method, and a matching plan for the task requirement data is obtained after the matching calculation is completed; Arrange the matching data in the real-time computing capacity strategy data, the matching data including node ID, available resources, current load and node status; Use the greedy algorithm to assign the matching plan to the node with the least load in the matching data; When the nodes in the matching data are overloaded, dynamic load balancing is used to reallocate some tasks to nodes with lower loads; After the matching plan is assigned to the matching data, the computing power task allocation data is obtained.

[0011] Preferably, a node communication mechanism is established for each node in the computing task allocation data in S4, including: Identify the node ID of each node in the computing task allocation data; Select the communication protocol based on the node ID, including TCP, UDP and MPI; After the communication protocol is selected, the parameters of the selected communication protocol are confirmed, including the port number, communication mode and data transmission format; After the parameters of the communication protocol are confirmed, a communication connection is established between the nodes, and the communication connection includes a heartbeat mechanism connection or a bidirectional connection; The heartbeat mechanism connection is to send heartbeat packets to each node at regular intervals and confirm the survival status of the node according to the heartbeat packet parameters; the bidirectional connection is to establish a two-way real-time transmission of tasks and data between nodes; After the communication connection is confirmed, the connection data between the receiving task and the computing power node in the computing network is obtained.

[0012] Preferably, setting the heartbeat packet sending time interval corresponding to the heartbeat mechanism includes: Extract the total number of nodes; Extract the data transmission time of the data packet of the unit data volume corresponding to each node, wherein the value range of the unit data volume is 10BM-100BM; Extract the data transmission rate corresponding to each node; Under the current data transmission environment corresponding to the node, the data transmission duration of the node for a data packet of unit data volume and the data transmission rate corresponding to each node are obtained to obtain the heartbeat packet sending time interval coefficient corresponding to each node; The heartbeat packet sending time interval coefficient corresponding to each node is obtained by the following formula: Among them, F represents the heartbeat packet sending time interval coefficient corresponding to each node; n represents the number of data transmission times corresponding to each node; B i represents the data transmission rate corresponding to the i-th data transmission of each node; B e T represents the data transmission rate of each node corresponding to the unit data volume received by the data packet; i represents the data transmission duration of the i-th data transmission corresponding to each node; T e Indicates the data transmission duration of the data packet per unit data volume corresponding to the node; x i represents the number of unit data contained in the data volume corresponding to the i-th data transmission of each node; T s Indicates the theoretical data transmission duration of a data packet per unit data volume corresponding to the maximum data transmission speed; Obtain the heartbeat packet sending time interval corresponding to all nodes according to the heartbeat packet sending time interval coefficient; The heartbeat packet sending time interval is obtained by the following formula: Where G represents the heartbeat packet sending time interval; G0 represents the preset initial heartbeat packet sending time interval; m represents the total number of nodes; F i F represents the heartbeat packet sending time interval coefficient corresponding to the i-th node; z F represents the median value of the heartbeat packet sending time interval coefficient corresponding to m nodes; cmax represents the maximum value of the coefficient difference of the heartbeat packet sending time interval between every two nodes with data transmission interactive connection corresponding to m nodes; F b represents the standard deviation of the heartbeat packet sending time interval coefficient corresponding to m nodes; F cb It represents the standard deviation of the coefficient difference of the heartbeat packet sending time interval between every two nodes with data transmission interactive connection; Send a heartbeat packet to each node according to the heartbeat packet sending time interval; The node survival rate and node resurrection rate corresponding to each heartbeat packet sent are monitored in real time, and the heartbeat packet sending time interval is adjusted according to the node survival rate and node resurrection rate.

[0013] Preferably, real-time monitoring of the node survival rate and node resurrection rate corresponding to each heartbeat packet sent, and adjusting the heartbeat packet sending time interval according to the node survival rate and node resurrection rate, include: Extract the node survival rate and node resurrection rate corresponding to each heartbeat packet sent; Comparing the node survival rate with a preset node survival rate threshold; When the node survival rate is lower than a preset node survival rate threshold, the heartbeat packet sending time interval is adjusted using the node survival rate and the node revival rate to obtain an adjusted heartbeat packet sending time interval; The adjusted heartbeat packet sending time interval is obtained by the following formula: Among them, G t represents the adjusted heartbeat packet sending time interval; G represents the heartbeat packet sending time interval; G0 represents the preset initial heartbeat packet sending time interval; k represents the number of heartbeat packet sending times before the node survival rate is lower than the preset node survival rate threshold; P c Indicates the node survival rate that is lower than the preset node survival rate threshold; P f Indicates the node resurrection rate when the node survival rate is lower than the preset node survival rate threshold; P ci represents the node survival rate corresponding to the sending of the i-th heartbeat packet; P firepresents the node revival rate corresponding to the sending of the i-th heartbeat packet; P cy Indicates the preset node survival rate threshold; A heartbeat packet is sent to each node according to the adjusted heartbeat packet sending time interval.

[0014] Preferably, data mapping is performed on the computing task allocation data and the computing node connection data in S5, and network layer fusion is performed after the data mapping is completed, including: Confirm the key fields in the computing task allocation data and the computing node connection data, where the key fields of the computing task allocation data include task ID, assigned node ID, task type, required resources, priority and data dependency; the key fields of the computing node connection data include node ID, available resource data, current load status, node status and communication information; Retrieve mapping rules from the database, and perform data mapping between computing task allocation data and computing node connection data according to the mapping rules; The data mapping process includes extracting task information from the computing task allocation data according to the task ID and the assigned node ID, and then finding the corresponding node data from the computing node connection data and comparing them; When the comparison result meets the mapping rule, the task is mapped to the node, and the task mapping data is obtained after the mapping is completed; When the comparison result does not satisfy the mapping rule, the mapping task is retried from the standby node or the task allocation strategy is re-evaluated until the comparison result satisfies the mapping rule.

[0015] Preferably, in S5, data mapping is performed on the computing task allocation data and the computing node connection data, and after the data mapping is completed, network layer fusion is performed, which also includes: The task mapping data is integrated at the network layer. Before the network layer integration, the task mapping data is first organized into a fusion data set. The fusion data set includes extracting the relevant information of each task from the mapped task mapping data, and extracting the latest status and resource utilization of each node from the computing power node connection data; According to the task execution status in the fused data set, the node data is updated. The node data includes the number of tasks assigned to each node, the current load and available resources. When the node load exceeds the preset range, dynamic adjustment is performed. The node data is updated or dynamically adjusted to perform data set integration, and after the data set integration, the standard computing power task allocation data is obtained.

[0016] Heterogeneous computing power integration and dynamic optimization allocation system, including: Task execution monitoring unit, used to: When the standard computing power task allocation data is being implemented, the standard computing power task allocation data is monitored in real time; Before real-time monitoring of standard computing task allocation data, the monitoring indicators are confirmed first. The monitoring indicators include task status, execution progress, resource usage, node load and abnormal events; During the implementation of the standard computing task allocation data, the monitoring indicators of each node are monitored in real time; Perform real-time data analysis of standard computing task allocation data based on the monitoring indicators of each node; Produce early warning prompts based on the analyzed real-time data, where the early warning prompt is that the threshold of the real-time data exceeds the preset range, and the real-time data exceeding the preset range is marked as early warning data; Adjust the warning data according to the degree of abnormality and generate a report; Finally, the warning data and adjustment report are transmitted to the display terminal for visual display.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. The heterogeneous computing power fusion and dynamic optimization allocation method and system provided by the present invention selects a communication protocol with a reliable transmission mechanism and error recovery capability, which can ensure the integrity and accuracy of data during transmission. Through security measures such as encryption and authentication, the security of data transmission can be further enhanced, and user privacy and data security can be protected. By adjusting the parameters and configuration of the communication protocol, it can easily adapt to computing power networks of different sizes and types, and use dynamic load balancing technology to reallocate some tasks to nodes with lower loads. The dynamic adjustment mechanism helps to maintain the stability and efficiency of the computing power network and avoid performance bottlenecks caused by single-point overload.

[0018] 2. The heterogeneous computing power fusion and dynamic optimization allocation method and system provided by the present invention realizes accurate mapping between data by clarifying the key fields in the computing power task allocation data and the computing power node connection data. This helps to ensure that tasks are correctly allocated to nodes with appropriate resources and status. After the data mapping is completed, network layer fusion is performed, which helps to achieve more efficient task execution and resource utilization in computing power allocation.

[0019] 3. The heterogeneous computing power fusion and dynamic optimization allocation method and system provided by the present invention perform real-time data analysis on the monitoring indicators of each node, which helps to achieve more refined computing power resource management. By analyzing the data, problems such as unbalanced resource use and excessive node load can be discovered, thereby providing strong support for optimizing computing power allocation and improving resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1It is a schematic diagram of the steps of heterogeneous computing power integration and dynamic optimization allocation of the present invention; Figure 2 It is a schematic diagram of the heterogeneous computing power integration and dynamic optimization allocation process of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In order to solve the problem that the existing technology does not adjust the load strategy according to the specific situation of the actual request task, which leads to the reduction of data load capacity, please refer to Figure 1 and Figure 2 , this embodiment provides the following technical solutions: The heterogeneous computing power integration and dynamic optimization allocation method includes the following steps: S1: Confirmation of central computing power: collect the hardware configuration information in the acquisition interface, process the collected hardware configuration information and extract key parameters, perform benchmark tests based on the extracted key parameters, and obtain real-time computing power data after the benchmark tests; Among them, by collecting hardware configuration information in detail and performing benchmark tests, potential performance issues or hardware failures can be discovered and resolved in a timely manner; S2: Load balancing strategy formulation: load analysis is performed based on real-time computing power data and standard computing power data, and load balancing strategy is formulated based on the load situation. After the load balancing strategy is formulated, the real-time computing power strategy data is obtained; Among them, through the formulation and implementation of load balancing strategies, computing resources can be reasonably allocated and scheduled to avoid resource waste and bottlenecks; S3: Node task allocation: Analyze the task requirements of the received task requests, and use the real-time computing power strategy data to allocate the analyzed tasks to the nodes. After the node task allocation is completed, the computing power task allocation data is obtained; Among them, the dynamic adjustment mechanism helps to maintain the stability and efficiency of the computing network and avoid performance bottlenecks caused by single-point overload; S4: Node information collaborative processing: Establish a node communication mechanism for each node in the computing power task allocation data, and obtain computing power node connection data after the communication mechanism is established; Among them, by adjusting the parameters and configuration of the communication protocol, it can easily adapt to computing networks of different sizes and types, and can intelligently schedule computing resources according to the performance and load of the nodes; S5: Processing task network fusion management: Map the computing power task allocation data with the computing power node connection data, and perform network layer fusion after the data mapping is completed. After the network layer fusion, the standard computing power task allocation data is obtained; Among them, the adjustability of the mapping rules enables the system to be customized according to actual needs, thereby improving the flexibility and accuracy of task allocation.

[0023] In step S1, the hardware configuration information in the acquisition interface is collected, and the collected hardware configuration information is processed and key parameters are extracted. Benchmark tests are performed based on the extracted key parameters, including: The hardware configuration information in the acquisition interface includes processor information, memory information, storage device information, network interface information, computing unit and resource usage information; After the hardware configuration information is collected, the key parameters of the hardware configuration information are extracted; Among them, the key parameters of processor information include the number of cores, number of threads, main frequency, maximum acceleration frequency and cache size; the key parameters of memory information include total capacity, type, speed and timing; the key parameters of storage device information include type, capacity, read and write speed and interface type; the key parameters of network interface information include type, speed, MAC address and IP address; the key parameters of computing unit include model, number of CUDA cores, video memory size, video memory type and computing power; the key parameters of resource usage information include COU usage, memory usage, disk frequency, disk speed and network traffic; The key parameters of the hardware configuration information are extracted and benchmarked; The test results are confirmed according to the benchmark test results, and the confirmed test results are used to generate real-time computing power data.

[0024] Specifically, by collecting hardware configuration information in detail through the acquisition interface and extracting key parameters, we can accurately understand the specific configuration and capabilities of each computing node, which is conducive to more refined resource allocation in computing power allocation, ensuring that tasks are assigned to the most appropriate computing nodes, thereby improving the utilization of computing resources and the efficiency of the overall computing power network. Benchmarking based on the key parameters of the hardware configuration information can evaluate the actual performance of each computing node. The benchmark test results can serve as an important basis for computing power allocation, ensuring that computing power resources can be dynamically adjusted and optimized according to task requirements to meet the needs of AI applications of different scales and types. By collecting hardware configuration information in detail and performing benchmarking, potential performance issues can be discovered and resolved in a timely manner. or hardware failure, which helps to improve the reliability and stability of the computing network, ensure that the task can be completed smoothly, and avoid computing interruptions or task failures caused by hardware problems. By continuously collecting and analyzing hardware configuration information and benchmark results, it can promote the continuous innovation and development of computing network technology, improve the overall computing power level and service quality, and accurately understand the performance and configuration of each computing node. More effective resource scheduling and allocation can be achieved to avoid resource waste. Among them, the processor is benchmarked using the CPU-Benchmark tool; the memory is benchmarked using the memtest tool; the storage device information is benchmarked using the CrystalDiskMark tool; the network interface is benchmarked using the Unigine Heaven tool; the computing unit is benchmarked using the 3DMark tool; and the resource usage monitoring tool is benchmarked.

[0025] In step S2, the load situation is analyzed based on the real-time computing power data and the standard computing power data, and a load balancing strategy is formulated based on the load situation, including: Retrieving standard computing capacity data from a database, where the standard computing capacity data is a standard expected performance indicator or a statistical average of historical performance data; Compare the real-time computing capability data with the performance indicators with the same attributes in the standard computing capability data, and calculate the difference value of each performance indicator based on the data comparison result; Perform load analysis based on the difference value; The load conditions are divided into normal load, light load and heavy load. When the percentage of the difference value is within the range of 80%-100%, it is a normal load; when the percentage of the difference value is less than 80%, it is a light load; when the percentage of the difference value is greater than 100%, it is a heavy load. A load analysis matrix is ​​constructed based on the analyzed load conditions, and load balancing operation marking is performed based on the constructed load matrix; Formulate load balancing strategies based on the load balancing operations marked in the load analysis matrix. Load balancing strategies include task migration, traffic regulation, and resource reallocation. After the load balancing strategy is formulated, real-time computing capacity strategy data is generated.

[0026] Specifically, by comparing the real-time computing power data with the standard computing power data, the current load situation can be quickly identified, so that adjustments can be made in time. The formulation and execution of load balancing strategies are based on accurate data analysis and judgment, ensuring the efficient use of resources. The use of statistical averages or standard expected performance indicators as standard computing power data provides a reliable benchmark for the comparison of real-time computing power data. By calculating the percentage of the difference value, the load situation (normal load, light load, heavy load) can be accurately divided, providing accurate information for the subsequent formulation of load balancing strategies. According to different load conditions, a variety of load balancing strategies including task migration, traffic regulation and resource reallocation are formulated to adapt to a variety of complex network environments and business needs. The flexibility of the load balancing strategy is also reflected in the ability to continuously adjust and optimize the strategy according to the generation of real-time computing power strategy data to adapt to the dynamic changes of the network environment. Through the formulation and execution of load balancing strategies, computing resources can be reasonably allocated and scheduled to avoid resource waste and bottlenecks. When the load is light, idle resources can be fully utilized through task migration or traffic regulation; when the load is heavy, resource reallocation can be used to reduce pressure and ensure the normal operation of the business. By building a load analysis matrix and marking load balancing operations, intelligent decision support is provided for the subsequent formulation of load balancing strategies.

[0027] Perform task demand analysis for the task request received in step S3, and use real-time computing capability strategy data to allocate the analyzed tasks to nodes, including: When the computing network receives a task request from an upper-layer application, it parses the task requirements of the task request; The parsed task requirements include task type, resource requirements, execution time, priority, and data dependencies; The task requirement data is matched and calculated using the task matching calculation method, and a matching plan for the task requirement data is obtained after the matching calculation is completed; Arrange the matching data in the real-time computing capacity strategy data, the matching data including node ID, available resources, current load and node status; Use the greedy algorithm to assign the matching plan to the node with the least load in the matching data; When the nodes in the matching data are overloaded, dynamic load balancing is used to reallocate some tasks to nodes with lower loads; After the matching plan is assigned to the matching data, the computing power task allocation data is obtained.

[0028] Specifically, a detailed task requirement analysis is performed on the received task request, including key information such as task type, resource requirements, execution time, priority, and data dependency. This comprehensive analysis helps ensure that subsequent task allocation is more accurate and efficient. Through the task matching calculation method, the solution can quickly generate a matching plan based on the task requirement data. Subsequently, the matching plan is assigned to the node with the smallest load using a greedy algorithm, which helps optimize the utilization of computing resources and reduce waiting time and computing costs. When the nodes in the matching data are overloaded, the solution can use dynamic load balancing technology to reallocate some tasks to nodes with lower loads. This dynamic adjustment mechanism helps maintain the stability and efficiency of the computing network and avoid performance bottlenecks caused by single-point overload. Through accurate task matching and allocation, as well as dynamic load balancing adjustment, the solution can significantly improve the utilization of computing resources. This helps reduce computing costs, improve overall computing efficiency, and dynamically adjust the allocation of computing resources according to changes in task requirements, thereby enhancing the flexibility and scalability of the system. This enables the computing network to better adapt to the needs of AI applications of different scales and types.

[0029] In order to solve the problem in the existing technology that the received tasks are not integrated with the current network layer in a targeted manner, resulting in poor computing power data integration, please refer to Figure 1 and Figure 2 , this embodiment provides the following technical solutions: A node communication mechanism is established for each node in the computing task allocation data in S4, including: Identify the node ID of each node in the computing task allocation data; Select the communication protocol based on the node ID, including TCP, UDP and MPI; After the communication protocol is selected, the parameters of the selected communication protocol are confirmed, including the port number, communication mode and data transmission format; After the parameters of the communication protocol are confirmed, a communication connection is established between the nodes, and the communication connection includes a heartbeat mechanism connection or a bidirectional connection; The heartbeat mechanism connection is to send heartbeat packets to each node at regular intervals and confirm the survival status of the node according to the heartbeat packet parameters; the bidirectional connection is to establish a two-way real-time transmission of tasks and data between nodes; After the communication connection is confirmed, the connection data between the receiving task and the computing power node in the computing network is obtained.

[0030] Specifically, by identifying the node ID of each node in the computing power task allocation data, the system can accurately identify and manage each node to ensure that computing power resources are efficiently utilized. It selects communication protocols (such as TCP, UDP, and MPI) according to the node ID and confirms the parameters of the communication protocol (such as port number, communication mode, and data transmission format). It can flexibly adapt to different application scenarios and computing power requirements. The establishment of the heartbeat mechanism connection enables the system to send heartbeat packets to each node at a regular interval to promptly confirm the survival status of the node and ensure the reliability and stability of computing power allocation. The bidirectional connection realizes the two-way real-time transmission of tasks and data between nodes, improves the response speed and flexibility of computing power allocation, and the heartbeat mechanism connection can promptly detect and handle node failures to ensure the stability and availability of the computing power network.

[0031] When a node fails, the system can quickly adjust the computing power allocation strategy and schedule tasks to other available nodes to avoid task interruption and data loss. The selected communication protocol has a reliable transmission mechanism and error recovery capability, which can ensure the integrity and accuracy of data during transmission. In addition, security measures such as encryption and authentication can further enhance the security of data transmission and protect user privacy and data security. By adjusting the parameters and configuration of the communication protocol, it can easily adapt to computing power networks of different sizes and types. It can intelligently schedule computing power resources according to the performance and load of the nodes to ensure that tasks are processed efficiently and evenly.

[0032] Specifically, setting the heartbeat packet sending time interval corresponding to the heartbeat mechanism includes: Extract the total number of nodes; Extract the data transmission time of the data packet of the unit data volume corresponding to each node, wherein the value range of the unit data volume is 10BM-100BM; Extract the data transmission rate corresponding to each node; Under the current data transmission environment corresponding to the node, the data transmission duration of the node for a data packet of unit data volume and the data transmission rate corresponding to each node are obtained to obtain the heartbeat packet sending time interval coefficient corresponding to each node; The heartbeat packet sending time interval coefficient corresponding to each node is obtained by the following formula: Among them, F represents the heartbeat packet sending time interval coefficient corresponding to each node; n represents the number of data transmission times corresponding to each node; B i represents the data transmission rate corresponding to the i-th data transmission of each node; B e T represents the data transmission rate of each node corresponding to the unit data volume received by the data packet; irepresents the data transmission duration of the i-th data transmission corresponding to each node; T e Indicates the data transmission duration of the data packet per unit data volume corresponding to the node; x i represents the number of unit data contained in the data volume corresponding to the i-th data transmission of each node; T s Indicates the theoretical data transmission duration of a data packet per unit data volume corresponding to the maximum data transmission speed; Obtain the heartbeat packet sending time interval corresponding to all nodes according to the heartbeat packet sending time interval coefficient; The heartbeat packet sending time interval is obtained by the following formula: Where G represents the heartbeat packet sending time interval; G0 represents the preset initial heartbeat packet sending time interval; m represents the total number of nodes; F i F represents the heartbeat packet sending time interval coefficient corresponding to the i-th node; z F represents the median value of the heartbeat packet sending time interval coefficient corresponding to m nodes; cmax represents the maximum value of the coefficient difference of the heartbeat packet sending time interval between every two nodes with data transmission interactive connection corresponding to m nodes; F b represents the standard deviation of the heartbeat packet sending time interval coefficient corresponding to m nodes; F cb It represents the standard deviation of the coefficient difference of the heartbeat packet sending time interval between every two nodes with data transmission interactive connection; Send a heartbeat packet to each node according to the heartbeat packet sending time interval; The node survival rate and node resurrection rate corresponding to each heartbeat packet sent are monitored in real time, and the heartbeat packet sending time interval is adjusted according to the node survival rate and node resurrection rate.

[0033] The technical effect of the above technical solution is: by comprehensively considering factors such as the number of nodes, data transmission rate and data transmission duration, the sending time interval of the heartbeat packet is dynamically adjusted, which can ensure that the network can avoid unnecessary heartbeat packet sending while maintaining a connection state, thereby improving network communication efficiency. The main purpose of the heartbeat mechanism is to monitor the survival status of the node. The scheme monitors the node survival rate and node resurrection rate in real time, and dynamically adjusts the heartbeat packet sending time interval based on this information, which helps to timely discover and handle node failures and enhance the stability and reliability of the network. Frequent sending of heartbeat packets will occupy network resources. By dynamically adjusting the heartbeat packet sending time interval, the scheme can reduce the sending frequency of heartbeat packets while ensuring network stability, thereby optimizing the utilization of network resources. The scheme takes into account the data transmission rate and data transmission duration of different nodes, and can dynamically adjust the heartbeat packet sending time interval according to the actual situation of each node. This enables the mechanism to adapt to different network environments and improve the flexibility and adaptability of network communication. By optimizing the sending strategy of the heartbeat packet, network delay and bandwidth occupancy can be reduced, thereby improving user experience. This optimization is particularly important in application scenarios with high real-time requirements.

[0034] In summary, this technical solution optimizes network communication efficiency, stability, resource utilization, network adaptability, and user experience by dynamically adjusting the heartbeat packet sending time interval. This helps to improve the overall performance of network communication and meet the growing network communication needs.

[0035] Specifically, real-time monitoring of the node survival rate and node resurrection rate corresponding to each heartbeat packet sent, and adjusting the heartbeat packet sending time interval according to the node survival rate and node resurrection rate, including: Extract the node survival rate and node resurrection rate corresponding to each heartbeat packet sent; Comparing the node survival rate with a preset node survival rate threshold; When the node survival rate is lower than a preset node survival rate threshold, the heartbeat packet sending time interval is adjusted using the node survival rate and the node revival rate to obtain an adjusted heartbeat packet sending time interval; The adjusted heartbeat packet sending time interval is obtained by the following formula: Among them, G t represents the adjusted heartbeat packet sending time interval; G represents the heartbeat packet sending time interval; G0 represents the preset initial heartbeat packet sending time interval; k represents the number of heartbeat packet sending times before the node survival rate is lower than the preset node survival rate threshold; P c Indicates the node survival rate that is lower than the preset node survival rate threshold; P fIndicates the node resurrection rate when the node survival rate is lower than the preset node survival rate threshold; P ci represents the node survival rate corresponding to the sending of the i-th heartbeat packet; P fi represents the node revival rate corresponding to the sending of the i-th heartbeat packet; P cy Indicates the preset node survival rate threshold; A heartbeat packet is sent to each node according to the adjusted heartbeat packet sending time interval.

[0036] The technical effect of the above technical solution is: by real-time monitoring of node survival rate and node resurrection rate, the solution can timely discover abnormal conditions in the network, such as node failure or network instability. When the node survival rate is lower than the preset threshold, by adjusting the heartbeat packet sending time interval, the node status can be monitored more frequently, so that timely measures can be taken to restore network connectivity and improve the reliability of network communication. When the network is stable and the node survival rate is high, reducing the sending frequency of heartbeat packets can save network resources. When the node survival rate decreases, by increasing the sending frequency of heartbeat packets, node failures can be detected more quickly, avoiding resources from being wasted on invalid communications. This dynamic adjustment mechanism helps to optimize the utilization of network resources. The solution dynamically adjusts the heartbeat packet sending time interval according to the actual survival rate and resurrection rate of the node, reflecting the adaptability of the system. This adaptability enables the system to automatically adjust the strategy according to changes in the network environment, so as to respond to various network conditions more flexibly. By timely detecting and handling node failures, the solution helps to reduce network interruptions and delays and improve user experience. This optimization is particularly important in application scenarios with high real-time requirements, such as online games, video conferencing, etc. By automatically adjusting the heartbeat packet sending interval, this solution can reduce the frequency of manual intervention and reduce the cost of network maintenance. At the same time, since network problems can be discovered and handled in a timely manner, it also helps to reduce downtime and losses caused by network failures.

[0037] In summary, this technical solution improves the reliability of network communication, optimizes resource utilization, enhances the adaptability of the system, improves user experience and reduces maintenance costs by dynamically adjusting the heartbeat packet sending time interval. These technical effects make this solution have broad application prospects in the field of network communication.

[0038] In S5, the computing task allocation data and the computing node connection data are mapped. After the data mapping is completed, the network layer is integrated, including: Confirm the key fields in the computing task allocation data and the computing node connection data, where the key fields of the computing task allocation data include task ID, assigned node ID, task type, required resources, priority and data dependency; the key fields of the computing node connection data include node ID, available resource data, current load status, node status and communication information; Retrieve mapping rules from the database, and perform data mapping between computing task allocation data and computing node connection data according to the mapping rules; The data mapping process includes extracting task information from the computing task allocation data according to the task ID and the assigned node ID, and then finding the corresponding node data from the computing node connection data and comparing them; When the comparison result meets the mapping rule, the task is mapped to the node, and the task mapping data is obtained after the mapping is completed; When the comparison result does not satisfy the mapping rule, the mapping task is retried from the standby node or the task allocation strategy is re-evaluated until the comparison result satisfies the mapping rule.

[0039] The task mapping data is integrated at the network layer. Before the network layer integration, the task mapping data is first organized into a fusion data set. The fusion data set includes extracting the relevant information of each task from the mapped task mapping data, and extracting the latest status and resource utilization of each node from the computing power node connection data; According to the task execution status in the fused data set, the node data is updated. The node data includes the number of tasks assigned to each node, the current load and available resources. When the node load exceeds the preset range, dynamic adjustment is performed. The node data is updated or dynamically adjusted to perform data set integration, and after the data set integration, the standard computing power task allocation data is obtained.

[0040] Specifically, by clarifying the key fields in the computing power task allocation data and the computing power node connection data, accurate mapping between the data is achieved. This helps to ensure that tasks are correctly assigned to nodes with appropriate resources and status. After the data mapping is completed, network layer fusion is performed, which helps to achieve more efficient task execution and resource utilization in computing power allocation. The adjustability of the mapping rules enables the system to be customized according to actual needs, thereby improving the flexibility and accuracy of task allocation. When the comparison result does not meet the mapping rule, the system can retry the mapping task from the backup node or re-evaluate the task allocation strategy, which enhances the adaptability and robustness of the system. The fused data set contains the latest data of task information and node status, which helps the system to grasp the network status in real time and make more informed decisions. When the node load exceeds the preset range, the system makes dynamic adjustments, which helps to prevent overload and performance degradation, while improving resource utilization. By integrating task mapping data and node data, the system can generate standard computing power task allocation data, which helps to achieve more efficient computing power allocation. Optimized computing power allocation can reduce resource waste and improve task execution efficiency, thereby meeting the computing power needs of more users.

[0041] In order to solve the problem in the prior art that when receiving a task and implementing it, the implementation process is not monitored in real time, resulting in the inability to quickly warn of abnormal data during the implementation process, please refer to Figure 1 and Figure 2 , this embodiment provides the following technical solutions: Heterogeneous computing power integration and dynamic optimization allocation system, including: Task execution monitoring unit, used to: When the standard computing power task allocation data is being implemented, the standard computing power task allocation data is monitored in real time; Before real-time monitoring of standard computing task allocation data, the monitoring indicators are confirmed first. The monitoring indicators include task status, execution progress, resource usage, node load and abnormal events; During the implementation of the standard computing task allocation data, the monitoring indicators of each node are monitored in real time; Perform real-time data analysis of standard computing task allocation data based on the monitoring indicators of each node; Produce early warning prompts based on the analyzed real-time data, where the early warning prompt is that the threshold of the real-time data exceeds the preset range, and the real-time data exceeding the preset range is marked as early warning data; Adjust the warning data according to the degree of abnormality and generate a report; Finally, the warning data and adjustment report are transmitted to the display terminal for visual display.

[0042] Specifically, by real-time monitoring of standard computing task allocation data, the system can instantly obtain key information such as task status, execution progress, resource usage, etc., so as to achieve effective management and scheduling of computing resources. This real-time monitoring mechanism helps to timely discover and solve potential problems, ensure that tasks can be completed on time and efficiently, and analyze real-time data according to preset thresholds. Once the data exceeds the range, an early warning prompt will be issued immediately. This early warning mechanism helps to identify and respond to possible risks in advance and prevent problems from expanding. The generation of early warning data and adjustment reports provides timely decision-making support for managers, helping them to take quick measures to reduce risks. Early warning data and adjustment reports are visualized through display terminals, allowing managers to intuitively understand the operating status and existing problems of the system. This visual display method not only improves the readability of information, but also reduces the difficulty of understanding and operation, allowing managers to make decisions faster and perform real-time data analysis on the monitoring indicators of each node, which helps to achieve more refined computing resource management. By analyzing the data, we can find problems such as unbalanced resource usage and excessive node load, thus providing strong support for optimizing computing power allocation and improving resource utilization. Real-time monitoring and early warning mechanisms help to promptly discover and repair faults and anomalies in the system, thereby improving system stability and reliability. This mechanism can also reduce losses caused by system crashes or data loss, and ensure business continuity and data integrity.

[0043] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0044] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.

Claims

1. Heterogeneous computing power integration and dynamic optimization allocation method, characterized in that: The steps include: S1: Confirmation of central computing power: collect the hardware configuration information in the acquisition interface, process the collected hardware configuration information and extract key parameters, perform benchmark tests based on the extracted key parameters, and obtain real-time computing power data after the benchmark tests; S2: Load balancing strategy formulation: load analysis is performed based on real-time computing power data and standard computing power data, and load balancing strategy is formulated based on the load situation. After the load balancing strategy is formulated, the real-time computing power strategy data is obtained; S3: Node task allocation: Analyze the task requirements of the received task requests, and use the real-time computing power strategy data to allocate the analyzed tasks to the nodes. After the node task allocation is completed, the computing power task allocation data is obtained; S4: Node information collaborative processing: Establish a node communication mechanism for each node in the computing power task allocation data, and obtain computing power node connection data after the communication mechanism is established; S5: Processing task network fusion management: Map the computing power task allocation data with the computing power node connection data, and perform network layer fusion after the data mapping is completed. After the network layer fusion, the standard computing power task allocation data is obtained.

2. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 1 is characterized in that: In step S1, the hardware configuration information in the acquisition interface is collected, and the collected hardware configuration information is processed and key parameters are extracted. Benchmark tests are performed based on the extracted key parameters, including: The hardware configuration information in the acquisition interface includes processor information, memory information, storage device information, network interface information, computing unit and resource usage information; After the hardware configuration information is collected, the key parameters of the hardware configuration information are extracted; Among them, the key parameters of processor information include the number of cores, number of threads, main frequency, maximum acceleration frequency and cache size; the key parameters of memory information include total capacity, type, speed and timing; the key parameters of storage device information include type, capacity, read and write speed and interface type; the key parameters of network interface information include type, speed, MAC address and IP address; the key parameters of computing unit include model, number of CUDA cores, video memory size, video memory type and computing power; the key parameters of resource usage information include COU usage, memory usage, disk frequency, disk speed and network traffic; The key parameters of the hardware configuration information are extracted and benchmarked; The test results are confirmed according to the benchmark test results, and the confirmed test results are used to generate real-time computing power data.

3. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 2 is characterized in that: In step S2, the load situation is analyzed based on the real-time computing power data and the standard computing power data, and a load balancing strategy is formulated based on the load situation, including: Retrieving standard computing capacity data from a database, where the standard computing capacity data is a standard expected performance indicator or a statistical average of historical performance data; Compare the real-time computing capability data with the performance indicators with the same attributes in the standard computing capability data, and calculate the difference value of each performance indicator based on the data comparison result; Perform load analysis based on the difference value; The load conditions are divided into normal load, light load and heavy load; when the percentage of the difference value is within the range of 80%-100%, it is a normal load; when the percentage of the difference value is less than 80%, it is a light load; when the percentage of the difference value is greater than 100%, it is a heavy load; A load analysis matrix is ​​constructed based on the analyzed load conditions, and load balancing operation marking is performed based on the constructed load matrix; Formulate load balancing strategies based on the load balancing operations marked in the load analysis matrix. Load balancing strategies include task migration, traffic regulation, and resource reallocation. After the load balancing strategy is formulated, real-time computing capacity strategy data is generated.

4. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 3 is characterized in that: Perform task demand analysis for the task request received in step S3, and use real-time computing capability strategy data to allocate the analyzed tasks to nodes, including: When the computing network receives a task request from an upper-layer application, it parses the task requirements of the task request; The parsed task requirements include task type, resource requirements, execution time, priority, and data dependencies; The task requirement data is matched and calculated using the task matching calculation method, and a matching plan for the task requirement data is obtained after the matching calculation is completed; Arrange the matching data in the real-time computing capacity strategy data, the matching data including node ID, available resources, current load and node status; Use the greedy algorithm to assign the matching plan to the node with the least load in the matching data; When the nodes in the matching data are overloaded, dynamic load balancing is used to reallocate some tasks to nodes with lower loads; After the matching plan is assigned to the matching data, the computing power task allocation data is obtained.

5. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 4 is characterized in that: A node communication mechanism is established for each node in the computing task allocation data in S4, including: Identify the node ID of each node in the computing task allocation data; Select the communication protocol based on the node ID, including TCP, UDP and MPI; After the communication protocol is selected, the parameters of the selected communication protocol are confirmed, including the port number, communication mode and data transmission format; After the parameters of the communication protocol are confirmed, a communication connection is established between the nodes, and the communication connection includes a heartbeat mechanism connection or a bidirectional connection; The heartbeat mechanism connection is to send heartbeat packets to each node at regular intervals and confirm the survival status of the node based on the heartbeat packet parameters; the bidirectional connection is to establish bidirectional real-time transmission of tasks and data between nodes; After the communication connection is confirmed, the connection data between the receiving task and the computing power node in the computing network is obtained.

6. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 5 is characterized in that: Setting the heartbeat packet sending time interval corresponding to the heartbeat mechanism includes: Extract the total number of nodes; Extract the data transmission time of the data packet of the unit data volume corresponding to each node, wherein the value range of the unit data volume is 10BM-100BM; Extract the data transmission rate corresponding to each node; Under the current data transmission environment corresponding to the node, obtain the data transmission duration of the node for a data packet of unit data volume and the data transmission rate corresponding to each node to obtain the heartbeat packet sending time interval coefficient corresponding to each node; The heartbeat packet sending time interval coefficient corresponding to each node is obtained by the following formula: Among them, F represents the heartbeat packet sending time interval coefficient corresponding to each node; n represents the number of data transmissions corresponding to each node; B i represents the data transmission rate corresponding to the i-th data transmission of each node; B e T represents the data transmission rate of each node corresponding to the unit data volume of the data packet received; i represents the data transmission duration of the i-th data transmission corresponding to each node; T e Indicates the data transmission duration of the data packet per unit data volume corresponding to the node; x i represents the number of unit data contained in the data volume corresponding to the i-th data transmission of each node; T s Indicates the theoretical data transmission duration of a data packet per unit data volume corresponding to the maximum data transmission speed; Obtain the heartbeat packet sending time interval corresponding to all nodes according to the heartbeat packet sending time interval coefficient; The heartbeat packet sending time interval is obtained by the following formula: Where G represents the heartbeat packet sending time interval; G0 represents the preset initial heartbeat packet sending time interval; m represents the total number of nodes; F i F represents the heartbeat packet sending time interval coefficient corresponding to the i-th node; z F represents the median value of the heartbeat packet sending time interval coefficient corresponding to m nodes; cmax represents the maximum value of the coefficient difference of the heartbeat packet sending time interval between every two nodes with data transmission interactive connection corresponding to m nodes; F b represents the standard deviation of the heartbeat packet sending time interval coefficient corresponding to m nodes; F cb It represents the standard deviation of the coefficient difference of the heartbeat packet sending time interval between every two nodes with data transmission interactive connection; Send a heartbeat packet to each node according to the heartbeat packet sending time interval; The node survival rate and node resurrection rate corresponding to each heartbeat packet sent are monitored in real time, and the heartbeat packet sending time interval is adjusted according to the node survival rate and node resurrection rate.

7. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 6 is characterized in that: Real-time monitoring of the node survival rate and node resurrection rate corresponding to each heartbeat packet sent, and adjusting the heartbeat packet sending time interval according to the node survival rate and node resurrection rate, including: Extract the node survival rate and node resurrection rate corresponding to each heartbeat packet sent; Comparing the node survival rate with a preset node survival rate threshold; When the node survival rate is lower than a preset node survival rate threshold, the heartbeat packet sending time interval is adjusted using the node survival rate and the node revival rate to obtain an adjusted heartbeat packet sending time interval; The adjusted heartbeat packet sending time interval is obtained by the following formula: Among them, G t represents the adjusted heartbeat packet sending time interval; G represents the heartbeat packet sending time interval; G0 represents the preset initial heartbeat packet sending time interval; k represents the number of heartbeat packet sending times before the node survival rate is lower than the preset node survival rate threshold; P c Indicates the node survival rate that is lower than the preset node survival rate threshold; P f Indicates the node resurrection rate when the node survival rate is lower than the preset node survival rate threshold; P ci represents the node survival rate corresponding to the sending of the i-th heartbeat packet; P fi represents the node revival rate corresponding to the sending of the i-th heartbeat packet; P cy Indicates the preset node survival rate threshold; A heartbeat packet is sent to each node according to the adjusted heartbeat packet sending time interval.

8. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 5 is characterized in that: In S5, the computing task allocation data and the computing node connection data are mapped. After the data mapping is completed, the network layer is integrated, including: Confirm the key fields in the computing task allocation data and the computing node connection data, where the key fields of the computing task allocation data include task ID, assigned node ID, task type, required resources, priority and data dependency; the key fields of the computing node connection data include node ID, available resource data, current load status, node status and communication information; Retrieve mapping rules from the database, and perform data mapping between computing task allocation data and computing node connection data according to the mapping rules; The data mapping process includes extracting task information from the computing task allocation data according to the task ID and the assigned node ID, and then finding the corresponding node data from the computing node connection data and comparing them; When the comparison result meets the mapping rule, the task is mapped to the node, and the task mapping data is obtained after the mapping is completed; When the comparison result does not satisfy the mapping rule, the mapping task is retried from the standby node or the task allocation strategy is re-evaluated until the comparison result satisfies the mapping rule.

9. The method for heterogeneous computing power integration and dynamic optimization allocation according to claim 8 is characterized in that: In S5, the computing task allocation data and the computing node connection data are mapped, and after the data mapping is completed, the network layer is integrated, which also includes: The task mapping data is integrated at the network layer. Before the network layer integration, the task mapping data is first organized into a fusion data set. The fusion data set includes extracting the relevant information of each task from the mapped task mapping data, and extracting the latest status and resource utilization of each node from the computing power node connection data; According to the task execution status in the fused data set, the node data is updated. The node data includes the number of tasks assigned to each node, the current load and available resources. When the node load exceeds the preset range, dynamic adjustment is performed. The node data is updated or dynamically adjusted to perform data set integration, and after the data set integration, the standard computing power task allocation data is obtained.

10. A heterogeneous computing power fusion and dynamic optimization allocation system, applied in the heterogeneous computing power fusion and dynamic optimization allocation method according to claim 9, characterized in that: include: Task execution monitoring unit, used to: When the standard computing power task allocation data is being implemented, the standard computing power task allocation data is monitored in real time; Before real-time monitoring of standard computing task allocation data, the monitoring indicators are confirmed first. The monitoring indicators include task status, execution progress, resource usage, node load and abnormal events; During the implementation of the standard computing task allocation data, the monitoring indicators of each node are monitored in real time; Perform real-time data analysis of standard computing task allocation data based on the monitoring indicators of each node; Produce early warning prompts based on the analyzed real-time data, where the early warning prompt is that the threshold of the real-time data exceeds the preset range, and the real-time data exceeding the preset range is marked as early warning data; Adjust the warning data according to the degree of abnormality and generate a report; Finally, the warning data and adjustment report are transmitted to the display terminal for visual display.

Images