A task allocation method for distributed data computing

Abstract

The application discloses a task allocation method for distributed data calculation, comprising the following steps: collecting task characteristics, node computing power, network transmission and multi-dimensional data sharding parameters to construct a basic data set through a distributed task monitoring calculation platform; analyzing node computing power parameters through a computing power redundancy balance control model to divide computing power levels and redundancy intervals; matching tasks and node computing power by using a lightweight edge collaborative allocation model to screen a candidate node set; completing data self-adaptive sharding and generating a sharding allocation scheme through deep annealing sharding scheduling optimization network; iteratively adjusting transmission paths and timing in combination with network transmission parameters; confirming a final scheme based on real-time feedback of node running states, and simultaneously, realizing precise adaptation and efficient execution of task allocation through feature vector construction and matching verification, sharding rule analysis and adaptive adjustment, transmission quality evaluation and dynamic optimization, and improving distributed calculation resource utilization and overall operation efficiency.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, and in particular to a task allocation method for distributed data computing. Background Technology

[0002] With the widespread application of distributed computing technology in big data processing, artificial intelligence training, and other scenarios, the exponential growth of data scale and the diversification of task types place higher demands on the accuracy of task allocation, resource utilization, and collaborative efficiency in distributed data computing. Currently, distributed systems suffer from problems such as dynamic fluctuations in node computing power, insufficient data sharding and node adaptability, and unstable network transmission latency. Traditional task allocation methods struggle to balance the dynamic matching of multi-dimensional parameters. Therefore, there is an urgent need for technical solutions that integrate dedicated management and control models, collaborative allocation models, and network optimization to achieve efficient adaptation between tasks and node resources, meeting the core requirements of real-time, balanced, and reliable task allocation in distributed computing scenarios.

[0003] Existing technologies have two significant drawbacks in the allocation of distributed data computing tasks: First, they lack dynamic balancing and control of node computing power redundancy and a collaborative adaptation mechanism for edge nodes. Task allocation is based solely on static computing power parameters, failing to fully consider node computing power fluctuations, differences in redundant computing power ranges, and the collaborative capabilities of edge nodes. This results in insufficient matching between tasks and node computing power, leading to resource waste where some nodes are overloaded while others are idle. Second, the coordination between data sharding scheduling and network transmission optimization is insufficient. Traditional sharding methods do not incorporate annealing optimization logic for adaptive adjustment, and transmission paths and timing planning are not dynamically adapted to network conditions. This results in a mismatch between data sharding and node carrying capacity, excessive transmission latency, and impacts the overall efficiency of distributed computing, making it difficult to meet the allocation requirements of large-scale, high-real-time distributed data computing tasks. Summary of the Invention

[0004] In order to overcome the shortcomings and deficiencies of existing technologies, this invention provides a task allocation method for distributed data computing.

[0005] The technical solution adopted in this invention is a task allocation method for distributed data computing, comprising the following steps: S1 collects task characteristic parameters, node computing power parameters, network transmission parameters and data sharding parameters in distributed data computing scenarios through a distributed task monitoring and computing platform to construct a multi-dimensional task allocation basic dataset. S2, based on the computing power redundancy balance management model, dynamically analyzes the node computing power parameters in the basic dataset, divides the node computing power level and marks the redundant computing power range of each node. S3 utilizes a lightweight edge collaborative allocation model to perform matching calculations between task feature parameters and node computing power levels, and initially screens a set of candidate nodes that meet the task's computing power requirements. S4, through deep annealing sharding scheduling optimization network, adaptively shards data sharding parameters are processed to generate a sharding allocation scheme corresponding to the candidate node set; S5, combined with network transmission parameters, iteratively adjusts the fragmentation allocation scheme to optimize the task transmission path and data interaction timing between candidate nodes; S6, based on the node running status parameters fed back in real time by the distributed task monitoring and computing platform, makes the final confirmation of the adjusted sharding allocation scheme and completes the task allocation for distributed data computing.

[0006] Furthermore, the expression for the computing power redundancy balancing control model is: ,in, This is the redundancy balancing coefficient for computing power. Let be the actual computing power parameters of the i-th node. Let i be the task load parameter for the i-th node. Let be the computing power weight coefficient of the i-th node. Let be the computing power fluctuation parameter of the i-th node. Let be the stability weight parameter of the i-th node. This is the redundancy adjustment coefficient. Let i be the task priority parameter for the i-th node. This represents the total number of nodes.

[0007] Furthermore, the expression for the lightweight edge collaborative allocation model is: ,in, For task node matching degree, For the feature parameters of the j-th task, Let J be the feature weight parameters for the j-th task. Let be the collaborative capability parameter of the k-th edge node. Let be the communication weight parameter for the k-th edge node. Let j be the delay requirement parameter for the j-th task. Let J be the reliability weight parameter for the j-th task. Let be the distance parameter for the k-th edge node. Let be the bandwidth parameter of the k-th edge node. This represents the total number of tasks. This represents the total number of edge nodes. It is a cooperative operation operator.

[0008] Furthermore, the expression for the deep annealing fragmentation scheduling optimization network is: ,in, To optimize the target value for partitioned scheduling, Let t be the size parameter of the t-th data partition. Let t be the importance weight parameter for the t-th data slice. This is the annealing coefficient. Let be the scheduling temperature change parameter for the t-th partition. Let be the scheduling adjustment parameters for the t-th partition. Let be the initial scheduling efficiency parameter for the t-th partition. Let be the target scheduling efficiency parameter for the t-th partition. This represents the total number of data shards.

[0009] Furthermore, the task status feedback model expression of the distributed task monitoring and computing platform is as follows: ,in, This is the node status feedback value. These are the real-time operating parameters for the s-th node. These are the baseline operating parameters for the s-th node. Let be the feedback weight parameter of the s-th node. Let be the task completion progress parameter for the s-th node. Let be the anomaly warning weight parameter for the s-th node. The total number of monitoring indicators. For gradient operation operators, .

[0010] Furthermore, the task allocation efficiency model expression for the distributed data computing is as follows: ,in, Assign efficiency coefficients to tasks. The execution efficiency parameter for the z-th assigned task. The weight parameter for assigning the z-th task. Let be the efficiency fluctuation parameter for the z-th task. Let z be the transmission fluctuation parameter for the z-th task. Let be the collaborative gain parameter for the z-th task. Let be the resource consumption parameter for the z-th task. To allocate the total number of tasks.

[0011] Further, S3 includes the following sub-steps: S31, extracting the computing power demand threshold, data processing type, task execution time constraints, and interaction dependencies from the task feature parameters to construct a task feature vector matrix; S32, quantizing and decomposing the computing power range corresponding to the node computing power level to generate the computing power supply vector of each node, including peak computing power, continuous computing power, and concurrent computing power calibration dimension data; S33, performing element-wise matching operation between the task feature vector matrix and the node computing power supply vector, calculating the cosine similarity and Euclidean distance between the vectors, and initially screening nodes with similarity higher than a set threshold and distance lower than a set threshold; S34, combining the node's historical task execution records to perform a secondary verification on the initially screened nodes, eliminating nodes with task conflicts and historical anomalies, and forming a candidate node set.

[0012] Further, S4 includes the following sub-steps: S41, analyzing the total data volume, data type, granularity requirements, and integrity constraints in the data sharding parameters to determine the basic rules and boundary conditions for sharding; S42, performing preliminary sharding processing on the data based on the initial temperature parameters and cooling rate parameters of the deep annealing sharding scheduling optimization network, generating several initial sharding units and recording the characteristic identifiers of each unit; S43, adjusting the initial sharding units through the network's iterative scheduling mechanism, merging excessively small sharding units and splitting excessively large sharding units, so that the size of each sharding unit is compatible with the computing power carrying capacity of the candidate nodes; S44, allocating the adjusted sharding units to the corresponding nodes according to the number of candidate node sets and the computing power proportion of each node, forming a sharding allocation scheme and marking the transmission order of each shard.

[0013] Further, S5 includes the following sub-steps: S51, collecting bandwidth utilization, transmission delay, packet loss rate, and link stability data from network transmission parameters to construct a network transmission quality evaluation matrix; S52, performing a traversal analysis of the transmission paths between candidate nodes based on the evaluation matrix to select the optimal transmission path with the lowest transmission delay and the lowest packet loss rate; S53, formulating differentiated transmission timing plans according to the size and priority of each fragment in the fragmentation allocation scheme, prioritizing the transmission of high-priority fragments and fragments with smaller data volumes; S54, monitoring network status changes in real time during transmission, and dynamically adjusting the transmission path and timing plan when network parameters exceed a set threshold to ensure the continuity and stability of fragmented transmission.

[0014] A distributed data computing task allocation method is disclosed, implemented through different units, including: a multi-dimensional parameter acquisition and analysis unit, a computing power redundancy balancing and management unit, a lightweight edge collaboration matching unit, a deep annealing sharding scheduling optimization unit, a network transmission adaptive adjustment unit, and a task allocation final confirmation unit. The multi-dimensional parameter acquisition and analysis unit establishes a data interaction connection with a distributed task monitoring and computing platform to collect and analyze task characteristics, node computing power, network transmission, and data sharding correlation parameters. The computing power redundancy balancing and management unit receives node computing power parameters output by the multi-dimensional parameter acquisition and analysis unit and divides node computing power levels and redundant computing power ranges. The lightweight edge collaboration matching unit interacts with multiple... The dimensional parameter acquisition and analysis unit and the computing power redundancy balancing control unit are connected to each other, and candidate nodes are selected based on task feature parameters and node computing power levels. The deep annealing sharding scheduling optimization unit is connected to the multi-dimensional parameter acquisition and analysis unit, and adaptively shards the data sharding parameters and generates a sharding allocation scheme. The network transmission adaptive adjustment unit receives the sharding allocation scheme output by the deep annealing sharding scheduling optimization unit and the network transmission parameters output by the multi-dimensional parameter acquisition and analysis unit, and optimizes the transmission path and timing. The task allocation final confirmation unit is connected to the distributed task monitoring and computing platform and the network transmission adaptive adjustment unit, and finalizes the task allocation scheme based on the node running status parameters.

[0015] Beneficial Effects: This invention proposes a task allocation method for distributed data computing. Through multi-model collaboration and dynamic optimization throughout the entire process, it achieves efficient and accurate allocation of distributed data computing tasks. Addressing the lack of dynamic balancing control of node computing power redundancy and edge collaboration adaptation in existing technologies, this invention uses a computing power redundancy balancing control model to dynamically analyze node computing power parameters, accurately classifying computing power levels and redundancy ranges. Combined with a lightweight edge collaboration allocation model, it achieves deep matching between tasks and nodes, avoiding resource waste from node overload and idle computing power, thus improving computing power resource utilization. Addressing the deficiency in the coordination between data sharding and network transmission, this invention achieves adaptive data sharding through deep annealing sharding scheduling optimization, adapting to node carrying capacity. Furthermore, it dynamically optimizes transmission paths and timings based on network transmission parameters, reducing transmission latency and ensuring data transmission stability. Simultaneously, a distributed task monitoring and computing platform is used to achieve full-process parameter collection and real-time feedback. Combined with the collaborative operation of a multi-unit architecture, a closed-loop control system is formed from parameter collection, computing power matching, sharding scheduling to transmission optimization and solution confirmation, comprehensively improving the balance, adaptability, and efficiency of task allocation, meeting the core requirements of large-scale distributed data computing. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method steps of the present invention; Figure 2 This is a diagram showing the unit composition for implementing the method of the present invention. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, a task allocation method for distributed data computing includes the following steps: S1 collects task characteristic parameters, node computing power parameters, network transmission parameters and data sharding parameters in distributed data computing scenarios through a distributed task monitoring and computing platform to construct a multi-dimensional task allocation basic dataset. Specifically, step S1 involves collecting parameters across all dimensions and building a basic dataset through a distributed task monitoring and computing platform. The platform relies on monitoring agent modules deployed on each node of the distributed computing cluster to collect four types of key parameters in real time. The task characteristic parameters include the data processing volume of a single task, the computational complexity level, the dependencies of task execution, and the task priority. Specifically, the data volume of each task ranges from 100MB to 50GB, the computational complexity is divided into 10 levels according to the number of instruction executions, the dependencies are identified through the data interaction links between tasks, and the priority is divided into 5 levels. The node computing power parameters include the number of CPU cores (4 to 64 cores), memory capacity (8GB to 512GB), GPU computing power (if configured), disk I / O rate (100MB / s to 2GB / s), and current computing power utilization (0% to 95%) of each computing node. The network transmission parameters involve the inter-node link bandwidth (100Mbps to 100Gbps), transmission latency (1ms to 50ms), packet loss rate (0% to 1%), and link stability coefficient. The data sharding parameters include the total number of data records to be processed (100,000 to 100 million records), the average size of data blocks, the threshold for the number of shards (5 to 100), and the shard integrity verification index. The platform standardizes the format of the collected parameters, removes outliers (such as computing power parameters that exceed reasonable ranges or network data with a packet loss rate greater than 1%), establishes an association index based on task ID and node ID, and finally constructs a structured multi-dimensional task allocation basic dataset to provide complete and accurate parameter support for subsequent model calculations. The parameter collection frequency for this process is set to once every 500 milliseconds to ensure the real-time and timeliness of the data.

[0019] S2, based on the computing power redundancy balance management model, dynamically analyzes the node computing power parameters in the basic dataset, divides the node computing power level and marks the redundant computing power range of each node. Specifically, step S2 dynamically analyzes the node computing power parameters in the basic dataset based on the computing power redundancy balance management model. First, it extracts core parameters such as the number of CPU cores, memory capacity, and current computing power utilization rate of each node. Through the model's built-in weight allocation mechanism, it assigns weight ratios of 0.6, 0.3, and 0.1 to CPU computing power, memory resources, and disk I / O capabilities, respectively, and calculates the comprehensive computing power value of each node. Then, based on the distribution range of the comprehensive computing power value, the nodes are divided into four levels: ultra-high computing power nodes (comprehensive computing power value ≥ 800), high computing power nodes (600 ≤ comprehensive computing power value < 800), medium computing power nodes (400 ≤ comprehensive computing power value < 600), and low computing power nodes (comprehensive computing power value < 400). When marking redundant computing power intervals, the model calculates the difference between the node's overall computing power value and the computing power requirement corresponding to the current task load, combined with the node's computing power fluctuation (the difference between the maximum and minimum computing power values ​​within the past 10 minutes), to determine the baseline value of redundant computing power for each node. This baseline value is then corrected based on the node's stability coefficient (calculated based on nearly 24 hours of fault-free operation, ranging from 0.7 to 1.0), ultimately defining the redundant computing power interval for each node. For example, the redundant computing power interval for ultra-high computing power nodes is typically between 200 and 400, while the redundant computing power interval for low computing power nodes is between 0 and 50. This dynamic resolution process is executed every 10 seconds, responding in real-time to changes in node computing power, ensuring the accuracy of computing power level classification and redundant computing power interval marking. This provides a precise computing power basis for subsequent task and node matching, avoiding resource adaptation deviations caused by static computing power assessments.

[0020] S3 utilizes a lightweight edge collaborative allocation model to perform matching calculations between task feature parameters and node computing power levels, and initially screens a set of candidate nodes that meet the task's computing power requirements. Specifically, step S3 utilizes a lightweight edge collaborative allocation model to perform matching calculations between task feature parameters and node computing power levels. First, it extracts key information from the task feature parameters, such as the computing power requirement threshold (determined based on task data volume and computational complexity, ranging from 100 to 700) and data processing type (CPU-intensive, memory-intensive, I / O-intensive). From the node computing power level, it extracts core data such as the comprehensive computing power value, redundant computing power range, and node type (edge ​​node or central node). The model employs a hierarchical matching strategy. The first layer performs preliminary screening based on the computing power requirement threshold and the node's comprehensive computing power value, retaining nodes with a comprehensive computing power value not lower than the task's computing power requirement threshold. The second layer matches data processing type with node hardware compatibility; for example, CPU-intensive tasks prioritize nodes with ≥16 CPU cores, and memory-intensive tasks prioritize nodes with ≥64GB of memory. The third layer combines edge collaborative capability parameters to calculate the collaborative matching degree between tasks and nodes. This matching degree comprehensively considers factors such as the physical distance between the node and the task initiator, communication latency, and historical collaboration success rate (the percentage of successful collaborations in the last 30 days). After three layers of matching operations, a preliminary set of candidate nodes that meet the task's computing power requirements is selected. The number of nodes in the set is controlled between 3 and 10. At the same time, the model assigns a matching priority score (ranging from 0 to 100 points) to each candidate node. The score is calculated based on computing power adaptability (weight 0.5), collaborative matching degree (weight 0.3), and historical execution efficiency (weight 0.2), providing a priority basis for subsequent sharding allocation. The response time of this matching operation process is controlled within 1 second, meeting the real-time requirements of distributed computing.

[0021] S4, through deep annealing sharding scheduling optimization network, adaptively shards data sharding parameters are processed to generate a sharding allocation scheme corresponding to the candidate node set; Specifically, step S4 uses deep annealing sharding scheduling to optimize the network and perform adaptive sharding processing on the data sharding parameters. First, it analyzes the total number of data entries, average data block size, sharding number threshold, and integrity verification index in the data sharding parameters to determine the basic constraints of sharding, namely, the data volume of each shard does not exceed the single batch processing limit of the node (determined according to the node's disk I / O rate and memory capacity, ranging from 1GB to 10GB), and the number of shards does not exceed the set threshold. The network is based on the core logic of the deep annealing algorithm, aiming at optimal fragment balance (data volume difference between fragments ≤10%) and processing efficiency. First, an initial fragmentation scheme is generated, and the number of initial fragments is determined by the ratio of the total data size to the upper limit of a single fragment. Then, iterative optimization is performed through simulated annealing: an initial temperature is set (corresponding to the fragmentation adjustment range, ranging from 5 to 10). In each iteration, the boundaries of some fragments are randomly adjusted, and the adjusted objective function value is calculated (a quantitative indicator that integrates fragment balance, node adaptability, and transmission efficiency). If the objective function value is better, the adjustment result is retained; if it is worse, the adjustment is accepted with a certain probability (decreasing as the temperature decreases). The temperature gradually decreases according to the set cooling rate (decreasing by 10% in each iteration) until the temperature drops to the termination threshold (0.5). After multiple rounds of iteration, a sharding allocation scheme corresponding to the candidate node set is generated. The scheme specifies the shard number, shard data volume, shard data type, and shard integrity check code for each candidate node. At the same time, the number of shards is allocated according to the computing power ratio of the candidate nodes. For example, nodes with ultra-high computing power are allocated 30% to 40% of the total number of shards, and nodes with low computing power are allocated 5% to 10% of the total number of shards, to ensure that the sharding allocation is adapted to the computing power of the nodes.

[0022] S5, combined with network transmission parameters, iteratively adjusts the fragmentation allocation scheme to optimize the task transmission path and data interaction timing between candidate nodes; Specifically, step S5 iteratively adjusts the fragment allocation scheme based on network transmission parameters. First, network transmission parameters such as link bandwidth, transmission delay, packet loss rate, and link stability coefficient between candidate nodes are collected to construct a transmission quality matrix between nodes. Matrix elements represent the transmission quality score of the corresponding node pair (ranging from 0 to 100, weighted by bandwidth, delay, and packet loss rate, with weights of 0.4, 0.4, and 0.2, respectively). Based on this matrix, the optimal selection of transmission paths for each fragment is analyzed. For each fragment, all possible transmission paths (direct links between candidate nodes and data storage nodes or indirect links via relay nodes) are traversed, and the path with the highest transmission quality score is selected as the initial transmission path. Subsequently, the data interaction timing is optimized. A timing plan is formulated based on fragment priority (determined by task priority and fragment importance, divided into three levels) and the bandwidth occupancy of the transmission path: high-priority fragments are allocated transmission bandwidth first, and transmission is initiated when the link bandwidth occupancy rate is below 60%; multiple fragments of the same node are transmitted sequentially in ascending order of fragment data volume to avoid bandwidth congestion caused by simultaneous transmission. During the iterative adjustment process, network transmission parameters are collected every 500 milliseconds. If the transmission delay of a certain link exceeds the threshold (20ms) or the packet loss rate exceeds 0.5%, the fragment transmission path corresponding to that link is dynamically adjusted, switching the fragment to a backup link with better transmission quality. At the same time, the transmission timing of the relevant fragments is adjusted to ensure that the transmission of each fragment does not interfere with each other. Through multiple rounds of iteration (10 to 20 rounds), a fragment allocation adjustment scheme with the lowest transmission delay and optimal stability is finally formed.

[0023] S6, based on the node running status parameters fed back in real time by the distributed task monitoring and computing platform, makes the final confirmation of the adjusted sharding allocation scheme and completes the task allocation for distributed data computing.

[0024] Specifically, step S6 confirms the adjusted sharding allocation scheme based on the real-time node operating status parameters fed back by the distributed task monitoring and computing platform. The platform collects the operating status parameters of each candidate node in real time, including CPU utilization, memory utilization, disk I / O activity, network bandwidth utilization, and node temperature (range 30℃ to 80℃), with a collection frequency of once every 200 milliseconds. First, the node operating status is verified for compliance, determining whether the CPU utilization of each node is below 85%, memory utilization is below 80%, disk I / O activity is below 70%, and temperature is below 75℃. If any parameter of a node exceeds the threshold, the node is marked as temporarily overloaded, sharding allocation to it is suspended, the redundant computing power range of the node is reassessed, and if necessary, its corresponding shards are adjusted to other candidate nodes. Subsequently, the overall feasibility of the sharding allocation scheme is verified. The overall system load balancing (standard deviation of the computing power utilization rate of each node, target value ≤10%) and total transmission latency (sum of transmission times of all shards, target value ≤10 seconds) after the scheme is implemented are calculated. If the load balancing or total transmission latency does not meet the target requirements, the scheme is fine-tuned. Adjustments include reallocating node ownership of some shards and optimizing transmission timing. After verification and validation, the platform generates the final task allocation instruction, which includes the shard allocation list, transmission path, timing plan, and execution parameters for each node. This instruction is sent to each candidate node through a distributed communication protocol. After receiving the instruction, each node starts the shard reception and computation process. The platform continuously monitors the node status and shard processing progress during task execution until all shards are processed, thus formally completing the task allocation for distributed data computation.

[0025] Preferably, the expression for the computing power redundancy balancing control model is: ,in, This is the redundancy balancing coefficient for computing power. Let be the actual computing power parameters of the i-th node. Let i be the task load parameter for the i-th node. Let be the computing power weight coefficient of the i-th node. Let be the computing power fluctuation parameter of the i-th node. Let be the stability weight parameter of the i-th node. This is the redundancy adjustment coefficient. Let i be the task priority parameter for the i-th node. This represents the total number of nodes.

[0026] Specifically, the computing power redundancy balance management model is used to accurately quantify the degree of balance of computing power redundancy among nodes in a distributed computing scenario, providing a core decision-making basis for node selection in task allocation. When implementing this model, the actual computing power parameter of the i-th node is collected in real time through the distributed task monitoring and computing platform, and the value corresponds to the quantified value of the comprehensive computing power of the node's CPU, memory, GPU and other hardware; the task load parameter is the total computing power required by the node for the tasks it has undertaken; the computing power weight coefficient is set according to the node's role in the distributed cluster, with the value ranging from 0.7 to 0.9 for core nodes and from 0.3 to 0.6 for edge nodes; the computing power fluctuation parameter is obtained by calculating the difference between the maximum and minimum actual computing power of the node in the past 5 minutes, reflecting the node's computing power stability; the stability weight parameter is determined based on the node's fault-free running time in the past 72 hours, with a value of 1.0 when the fault-free running time exceeds 70 hours and a value of 0.7 when it is less than 24 hours; the redundancy adjustment coefficient is set by the system administrator according to the overall computing power redundancy of the cluster, with a value ranging from 0.5 to 1.5; the task priority parameter is divided into 1 to 5 levels according to the importance of the task, corresponding to quantified values ​​of 1 to 5; the total number of nodes is the total number of effective nodes participating in task allocation in the distributed cluster. The model obtains a computing power redundancy balance coefficient through comprehensive calculation. The coefficient ranges from 0 to 10. The closer the coefficient is to 10, the more balanced the distribution of computing power redundancy among nodes. A coefficient lower than 3 indicates a significant computing power redundancy imbalance. During implementation, the coefficient is recalculated every 15 seconds to dynamically adapt to changes in node computing power and task load, ensuring efficient utilization of node computing power resources and avoiding situations where some nodes are overloaded and others are idle.

[0027] Preferably, the expression for the lightweight edge collaborative allocation model is: ,in, For task node matching degree, For the feature parameters of the j-th task, Let J be the feature weight parameters for the j-th task. Let be the collaborative capability parameter of the k-th edge node. Let be the communication weight parameter for the k-th edge node. Let j be the delay requirement parameter for the j-th task. Let J be the reliability weight parameter for the j-th task. Let be the distance parameter for the k-th edge node. Let be the bandwidth parameter of the k-th edge node. This represents the total number of tasks. This represents the total number of edge nodes. It is a cooperative operation operator.

[0028] Specifically, the lightweight edge collaborative allocation module achieves precise matching between task characteristics and edge node collaborative capabilities, improving the adaptability and execution efficiency of task allocation. During implementation, the characteristic parameters of the j-th task include quantitative indicators such as task data volume, computational complexity, and latency requirements, obtained by integrating the configuration information at the time of task submission with the data collected by the distributed task monitoring and computing platform. The characteristic weight parameters are set according to the task type: a computational complexity weight of 0.6 for CPU-intensive tasks and a data volume weight of 0.7 for data-intensive tasks. The collaborative capability parameters of the k-th edge node are calculated comprehensively based on indicators such as the node's historical data interaction success rate with other nodes and communication response speed, with a value ranging from 0 to 100. The communication weight parameters are determined based on the node's network interface specifications and link bandwidth: 0.8 to 1.0 for nodes supporting 10 Gigabit Ethernet and 0.8 to 1.0 for nodes supporting Gigabit Ethernet. The node value ranges from 0.4 to 0.7; the task latency requirement parameter is the quantified value of the maximum allowable execution latency for the task, with smaller values ​​for tasks with high real-time requirements and larger values ​​for non-real-time tasks; the reliability weight parameter is set according to the importance of the task execution result, with values ​​ranging from 0.8 to 1.0 for core business tasks and from 0.5 to 0.7 for ordinary business tasks; the edge node distance parameter is the quantified value of the physical distance between the node and the task data storage node; the bandwidth parameter is the actual available bandwidth of the node network link; the total number of tasks is the total number of tasks currently to be assigned, and the total number of edge nodes is the total number of edge nodes in the cluster with task processing capabilities; the collaborative operation operator is the fusion operation rule built into the model, used to integrate the results of the two core calculations. The matching degree of the task nodes generated by the model calculation ranges from 0 to 100. A matching degree higher than 80 is the optimal matching node, and a matching degree lower than 50 is excluded as a candidate node. During the implementation process, the matching degree between each task to be assigned and all edge nodes is calculated in real time. Candidate nodes are selected by sorting them in descending order of matching degree to ensure that the task is assigned to the edge node with the most suitable collaborative capability. At the same time, the model calculation process is optimized to the millisecond level response to meet the real-time requirements of large-scale task assignment.

[0029] Preferably, the expression for the deep annealing fragmentation scheduling optimization network is: ,in, To optimize the target value for partitioned scheduling, Let t be the size parameter of the t-th data partition. Let t be the importance weight parameter for the t-th data slice. This is the annealing coefficient. Let be the scheduling temperature change parameter for the t-th partition. Let be the scheduling adjustment parameters for the t-th partition. Let be the initial scheduling efficiency parameter for the t-th partition. Let be the target scheduling efficiency parameter for the t-th partition. This represents the total number of data shards.

[0030] Specifically, the deep annealing sharding scheduling optimization network adaptively shards distributed data, generating a sharding allocation scheme that matches the candidate node set, ensuring the rationality of data sharding and transmission execution efficiency. During implementation, the size parameter of the t-th data shard is the quantified value of the data volume after sharding, with an initial value determined based on the total data size and the estimated number of shards; the importance weight parameter is set according to the degree of influence of the sharded data on the task execution result, with core data shards taking values ​​from 0.8 to 1.0 and auxiliary data shards taking values ​​from 0.3 to 0.6; the annealing coefficient is the core control parameter of the deep annealing algorithm, with an initial value ranging from 5 to 10, gradually adjusted during the iteration process; the sharding scheduling temperature change parameter reflects the decrease in temperature during algorithm iteration, initially... The initial value is relatively large during iteration, and then decreases by a fixed proportion thereafter. The scheduling adjustment parameter controls the magnitude of the sharding adjustment, ranging from 0.1 to 0.5, and dynamically adapts to the shard size and node capacity. The initial scheduling efficiency parameter is the estimated execution efficiency value of the initial sharding allocation scheme, calculated based on the shard size and the historical processing speed of the nodes. The target scheduling efficiency parameter is the preset optimal sharding scheduling efficiency quantification value, set according to the overall cluster performance. The total number of data shards is the final number of valid data shards generated, ranging from 5 to 100. During model implementation, several sharding schemes are first generated based on the initial parameters, and then optimized through multiple rounds of iteration. In each iteration, the temperature is adjusted according to the scheduling temperature change parameter, and the sharding scheme is corrected in conjunction with the scheduling adjustment parameter until the requirements corresponding to the target scheduling efficiency parameter are met. The generated sharding scheduling optimization target value ranges from 0 to 100, with higher values ​​indicating better sharding schemes. This process ensures that the size and importance of each shard match the computing power capacity and processing priority of the candidate nodes, improving the overall efficiency of sharding transmission and processing.

[0031] Preferably, the task status feedback model expression of the distributed task monitoring and computing platform is: ,in, This is the node status feedback value. These are the real-time operating parameters for the s-th node. These are the baseline operating parameters for the s-th node. Let be the feedback weight parameter of the s-th node. Let be the task completion progress parameter for the s-th node. Let be the anomaly warning weight parameter for the s-th node. The total number of monitoring indicators. For gradient operation operators, .

[0032] Specifically, the task status feedback model of the distributed task monitoring and computing platform is used to fully integrate multi-dimensional operating status parameters of nodes to generate accurate node status feedback values, providing a basis for adjusting and confirming task allocation schemes. During implementation, the real-time operating parameters of the s-th node include real-time data collection such as CPU utilization, memory utilization, disk I / O rate, and network bandwidth utilization, collected every 200 milliseconds. The baseline operating parameters are the standard values ​​of various operating parameters of the node under normal load, determined by the node's factory performance indicators and system configuration parameters. The feedback weight parameters are set according to the degree of influence of each operating parameter on the node status, with the weights of CPU utilization and memory utilization ranging from 0.3 to 0.4, and the weights of disk I / O rate and network bandwidth utilization ranging from 0.1 to 0.2. The task completion progress parameter is the completion ratio of the node's current tasks, ranging from 0 to 1.0. The anomaly warning weight parameter is used to strengthen the feedback of abnormal states, ranging from 1.2 to 1.5. When a certain operating parameter of the node approaches a threshold, this parameter can amplify the impact of the abnormal state. The total number of monitoring indicators is the total number of parameter types participating in the node status assessment, typically including 8 to 12 core indicators. The model performs gradient analysis on parameters related to task completion progress using gradient operation operators to capture the changing trends of task execution progress. Then, it integrates the calculation results of various operating parameters through multi-dimensional state fusion operators to finally generate node status feedback values. The feedback values ​​range from 0 to 10, where 0 to 3 indicates an abnormal node status, 3 to 7 indicates a normal node status, and 7 to 10 indicates an excellent node status. During implementation, node status feedback values ​​are generated every 500 milliseconds to provide real-time feedback on node operating status, providing timely and accurate basis for dynamic adjustment of task allocation schemes.

[0033] Preferably, the task allocation efficiency model expression for distributed data computing is: ,in, Assign efficiency coefficients to tasks. The execution efficiency parameter for the z-th assigned task. The weight parameter for assigning the z-th task. Let be the efficiency fluctuation parameter for the z-th task. Let z be the transmission fluctuation parameter for the z-th task. Let be the collaborative gain parameter for the z-th task. Let be the resource consumption parameter for the z-th task. To allocate the total number of tasks.

[0034] Specifically, the distributed data computing task allocation efficiency model is used to quantitatively evaluate the overall efficiency of task allocation schemes, providing a quantitative standard for scheme optimization and confirmation. During implementation, the execution efficiency parameter of the z-th allocated task is the quantified value of the actual execution speed of the task on the allocation node, collected in real time through a distributed task monitoring and computing platform; the weight parameter is set according to the task's priority and resource consumption, with high-priority, high-resource-consumption tasks having a weight of 0.7 to 0.9, and low-priority, low-resource-consumption tasks having a weight of 0.3 to 0.6; the efficiency fluctuation parameter is the difference between the maximum and minimum efficiency during task execution, reflecting the stability of task execution efficiency; the transmission fluctuation parameter is obtained by calculating the difference between the maximum and minimum transmission rate during task data transmission; the collaborative gain parameter is the quantified value of the efficiency improvement brought about by the collaborative execution of the task, nodes, and other related tasks, derived based on historical execution data statistics, with a value range of 0 to 2; the resource consumption parameter is the comprehensive quantified value of CPU, memory, network, and other resources consumed during task execution; and the total number of allocated tasks is the total number of tasks involved in the current allocation scheme. The model obtains a task allocation efficiency coefficient through comprehensive calculation. The coefficient ranges from 0 to 10. The higher the coefficient, the higher the overall efficiency of the task allocation scheme. When the coefficient is lower than 4, the allocation scheme needs to be optimized and adjusted. During implementation, the initial calculation is performed after the task allocation scheme is initially determined and before final execution. The calculation is repeated every 30 seconds during task execution. The task allocation scheme is dynamically adjusted according to the changes in the coefficient. By optimizing the node allocation, transmission path and timing of tasks, the overall efficiency of task allocation is improved, resource consumption is reduced, and the efficient completion of distributed data computing tasks is ensured.

[0035] Preferably, step S3 includes the following sub-steps: S31, extracting the computing power demand threshold, data processing type, task execution time constraints, and interaction dependencies from the task feature parameters to construct a task feature vector matrix; S32, quantizing and decomposing the computing power range corresponding to the node computing power level to generate the computing power supply vector for each node, including peak computing power, continuous computing power, and concurrent computing power calibration dimension data; S33, performing element-wise matching operations between the task feature vector matrix and the node computing power supply vector to calculate the cosine similarity and Euclidean distance between the vectors, and initially screening nodes with similarity higher than a set threshold and distance lower than a set threshold; S34, combining the node's historical task execution records to perform a secondary verification on the initially screened nodes, eliminating nodes with task conflicts and historical anomalies to form a candidate node set.

[0036] Specifically, step S3 achieves precise matching between task features and node computing power through four sub-steps, ensuring the rationality of the candidate node set. S31 first extracts four core information categories from the task feature parameters collected by the distributed task monitoring and computing platform: computing power requirement threshold, data processing type, task execution duration constraint, and interaction dependency relationship. The computing power requirement threshold is quantified into specific numerical ranges based on task data volume and computational complexity; the data processing type is clearly divided into three categories: CPU-intensive, memory-intensive, and I / O-intensive; the task execution duration constraint is set to a specific time interval; and the interaction dependency relationship is identified through the data flow direction between tasks. Based on this information, a structured task feature vector matrix is ​​constructed, with the matrix dimensions corresponding one-to-one with the extracted feature categories, ensuring the complete presentation of feature information. S32 quantifies and decomposes the computing power ranges corresponding to the node computing power levels defined by the computing power redundancy balancing management model. It breaks down the computing power range of each level into key dimensions such as peak computing power, sustained computing power, and concurrent computing power. Peak computing power is the maximum computing power a node can provide in a short period; sustained computing power is the computing power a node can maintain stable operation over a long period; and concurrent computing power is the computing power a node can process multiple tasks simultaneously. This quantization and decomposition generates computing power supply vectors for each node, making the node computing power information more readable and comparable. S33 employs a vector matching algorithm to perform element-wise matching operations between the task feature vector matrix and the node computing power supply vectors. It calculates the cosine similarity and Euclidean distance between the vectors, setting a cosine similarity threshold of 0.75 and an Euclidean distance threshold of 0.3. Nodes with a similarity higher than 0.75 and a distance lower than 0.3 are initially selected to ensure that the selected nodes basically meet the task computing power requirements. S34 retrieves the historical task execution records of nodes over the past 30 days, and calculates indicators such as task completion rate and number of abnormal interruptions. It then performs a second verification on the initially screened nodes, eliminating nodes with a task completion rate of less than 90% or more than 3 abnormal interruptions in the past 30 days that have task conflicts or historical anomalies. Finally, a set of candidate nodes is formed, providing a high-quality node resource foundation for subsequent sharding allocation. The entire process ensures the accuracy and reliability of the matching results.

[0037] Preferably, step S4 includes the following sub-steps: S41, analyzing the total data volume, data type, granularity requirements, and integrity constraints in the data sharding parameters to determine the basic rules and boundary conditions for sharding; S42, performing preliminary sharding processing on the data based on the initial temperature parameters and cooling rate parameters of the deep annealing sharding scheduling optimization network, generating several initial sharding units and recording the characteristic identifiers of each unit; S43, adjusting the initial sharding units through the network's iterative scheduling mechanism, merging excessively small sharding units and splitting excessively large sharding units, so that the size of each sharding unit is compatible with the computing power carrying capacity of the candidate nodes; S44, allocating the adjusted sharding units to the corresponding nodes according to the number of candidate node sets and the computing power proportion of each node, forming a sharding allocation scheme and marking the transmission order of each shard.

[0038] Specifically, step S4 achieves adaptive data sharding and reasonable allocation through four sub-steps, generating a sharding allocation scheme that adapts to the candidate node set. Step S31 first parses the data sharding parameters collected by the distributed task monitoring and computing platform, extracting four key elements: total data volume, data type, sharding granularity requirements, and integrity constraints. The total data volume is defined by the specific number of data entries or bytes; the data type is classified as structured, semi-structured, or unstructured; the sharding granularity requirements set the upper and lower limits of the data volume in a single shard; and the integrity constraints clarify the logical integrity requirements that the data must maintain after sharding. Based on these parsing results, the basic rules and boundary conditions for sharding are determined to ensure that the sharding process is systematic and predictable. S42 optimizes the initial temperature and cooling rate parameters of the network based on deep annealing and partitioning scheduling. The initial temperature parameter is set to a value range of 5 to 10, and the cooling rate parameter is set to decrease by a fixed percentage of 10% per iteration. The data is initially partitioned according to basic rules and boundary conditions, dividing the overall data into several initial partition units. Each partition unit is marked with a unique feature identifier, including key information such as data type, partition number, and data range, facilitating subsequent tracking and management. S43 initiates the network's iterative scheduling mechanism, gradually reducing the temperature parameter according to the cooling rate. The initial partition units are dynamically adjusted, merging excessively small partition units with data volumes below the lower limit of the partition granularity and splitting excessively large partition units with data volumes above the upper limit of the partition granularity. This ensures that the size of each partition unit after adjustment is within the required partition granularity range and is compatible with the computing power capacity of candidate nodes, avoiding mismatches between partition size and node computing power. S44 counts the number of candidate nodes and the computing power ratio of each node, allocates the number of shards according to the computing power ratio, and allocates more shards to nodes with higher computing power ratios. The adjusted shard units are then allocated to the corresponding nodes according to the allocated number. At the same time, the transmission order of each shard is marked according to the shard feature identifier and the node computing power, and shards with high compatibility with the nodes are transmitted first, thus forming a complete shard allocation scheme, laying the foundation for subsequent transmission optimization.

[0039] Preferably, step S5 includes the following sub-steps: S51, collecting bandwidth utilization, transmission delay, packet loss rate, and link stability data from network transmission parameters to construct a network transmission quality evaluation matrix; S52, performing a traversal analysis of the transmission paths between candidate nodes based on the evaluation matrix to select the optimal transmission path with the lowest transmission delay and the lowest packet loss rate; S53, formulating differentiated transmission timing plans according to the size and priority of each fragment in the fragmentation allocation scheme, prioritizing the transmission of high-priority fragments and fragments with smaller data volumes; S54, monitoring network status changes in real time during transmission, and dynamically adjusting the transmission path and timing plan when network parameters exceed a set threshold to ensure the continuity and stability of fragmented transmission.

[0040] Specifically, step S5 iteratively adjusts the fragmentation allocation scheme through four sub-steps, optimizing transmission paths and timing to ensure efficient and stable data transmission. S51 uses a distributed task monitoring and computing platform to collect real-time network transmission parameters such as bandwidth utilization, transmission latency, packet loss rate, and link stability data. The collection frequency is set to once every 500 milliseconds. Bandwidth utilization is recorded as a percentage, transmission latency as milliseconds, and packet loss rate as a percentage. Link stability data is quantified by the number of link interruptions within the past minute. Based on this collected data, a network transmission quality evaluation matrix is ​​constructed. The matrix rows and columns represent candidate nodes, and the elements are the transmission quality scores between corresponding nodes. The scores are calculated according to the weighted proportions of bandwidth utilization, transmission latency, packet loss rate, and link stability (0.3, 0.4, 0.2, and 0.1, respectively). S52 traverses and analyzes all possible transmission paths between candidate nodes based on the evaluation matrix, calculates the total transmission quality score for each path, and selects the optimal transmission path with the highest total transmission quality score (i.e., the lowest transmission latency and packet loss rate). Simultaneously, a backup path is marked for each optimal path for switching when network conditions change. S53, based on the size and priority of each fragment in the fragmentation allocation scheme (priority is divided into three levels according to task importance: high-priority fragments are marked as Level 1, medium-priority as Level 2, and low-priority as Level 3), formulates a differentiated transmission timing plan. Level 1 fragments are allocated transmission bandwidth first and transmission begins when the link bandwidth utilization rate is below 60%. Level 2 and Level 3 fragments are postponed accordingly. Simultaneously, within the same priority level, fragments are transmitted in ascending order of data size to avoid congestion caused by excessive bandwidth consumption from large fragment transmissions. S54 continuously monitors network status changes during transmission. When network parameters such as transmission delay exceeding 20 milliseconds, packet loss rate exceeding 0.5%, or bandwidth utilization exceeding 85% are detected to exceed set thresholds, a path and timing adjustment mechanism is immediately triggered. Fragments transmitted on that link are switched to backup paths, and the transmission timing of relevant fragments is adjusted to ensure the continuity and stability of fragment transmission. Through multiple rounds of iterative adjustments, the fragmentation allocation scheme achieves optimal transmission performance.

[0041] like Figure 2As shown, a distributed data computing task allocation method is implemented through different units, including: a multi-dimensional parameter acquisition and analysis unit, a computing power redundancy balancing and management unit, a lightweight edge collaborative matching unit, a deep annealing sharding scheduling optimization unit, a network transmission adaptive adjustment unit, and a task allocation final confirmation unit. The multi-dimensional parameter acquisition and analysis unit establishes a data interaction connection with the distributed task monitoring and computing platform to collect and analyze task characteristics, node computing power, network transmission, and data sharding correlation parameters. The computing power redundancy balancing and management unit receives the node computing power parameters output by the multi-dimensional parameter acquisition and analysis unit and divides the node computing power level and redundant computing power range. The lightweight edge collaborative matching unit... Connected to the multi-dimensional parameter acquisition and analysis unit and the computing power redundancy balancing control unit, the unit completes candidate node screening based on task characteristic parameters and node computing power levels. The deep annealing sharding scheduling optimization unit is connected to the multi-dimensional parameter acquisition and analysis unit, performs adaptive sharding processing on data sharding parameters, and generates a sharding allocation scheme. The network transmission adaptive adjustment unit receives the sharding allocation scheme output by the deep annealing sharding scheduling optimization unit and the network transmission parameters output by the multi-dimensional parameter acquisition and analysis unit, respectively, and optimizes the transmission path and timing. The task allocation final confirmation unit is connected to the distributed task monitoring and computing platform and the network transmission adaptive adjustment unit, and completes the final confirmation of the task allocation scheme based on the node running status parameters.

[0042] A task allocation method for distributed data computing addresses the lack of dynamic balancing of node computing power redundancy and edge collaborative adaptation in existing technologies. It uses a computing power redundancy balancing management model to dynamically analyze node computing power parameters, accurately classify computing power levels and redundancy ranges, and utilizes a lightweight edge collaborative allocation model to achieve deep matching between task characteristics and node computing power. It also performs secondary verification by combining node historical execution records, effectively avoiding resource waste caused by node overload and idle computing power, and significantly improving the utilization rate of computing power resources and the accuracy of task matching.

[0043] Furthermore, this method possesses the significant advantage of coordinated data sharding scheduling and transmission optimization, successfully addressing the insufficient coordination between data sharding and network transmission in traditional technologies. Through deep annealing and sharding scheduling optimization, the network adaptively shards data, ensuring precise matching between shard size and candidate node capacity. A quality assessment matrix is ​​then constructed based on network transmission parameters to select the optimal transmission path and formulate differentiated timing plans, dynamically adjusting transmission strategies in real time to address network state changes. Simultaneously, relying on a distributed task monitoring and computing platform, the entire process of parameter collection and real-time feedback is achieved. Combined with the closed-loop operation of a multi-unit architecture, a complete control chain is formed from parameter parsing, computing power matching, sharding scheduling to transmission optimization and solution confirmation, comprehensively improving the balance, stability, and efficiency of task allocation and meeting the complex needs of large-scale distributed data computing.

[0044] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A task allocation method for distributed data computing, characterized in that, Includes the following steps: S1 collects task characteristic parameters, node computing power parameters, network transmission parameters and data sharding parameters in distributed data computing scenarios through a distributed task monitoring and computing platform to construct a multi-dimensional task allocation basic dataset. S2, based on the computing power redundancy balance management model, dynamically analyzes the node computing power parameters in the basic dataset, divides the node computing power level and marks the redundant computing power range of each node. S3 utilizes a lightweight edge collaborative allocation model to perform matching calculations between task feature parameters and node computing power levels, and initially screens a set of candidate nodes that meet the task's computing power requirements. S4, through deep annealing sharding scheduling optimization network, adaptively shards the data sharding parameters and generates a sharding allocation scheme corresponding to the candidate node set; S5, combined with network transmission parameters, iteratively adjusts the fragmentation allocation scheme to optimize the task transmission path and data interaction timing between candidate nodes; S6, based on the node running status parameters fed back in real time by the distributed task monitoring and computing platform, makes the final confirmation of the adjusted sharding allocation scheme and completes the task allocation for distributed data computing.

2. The task allocation method for distributed data computing according to claim 1, characterized in that, The expression for the computing power redundancy balancing control model is: ,in, This is the redundancy balancing coefficient for computing power. Let be the actual computing power parameters of the i-th node. Let i be the task load parameter for the i-th node. Let be the computing power weight coefficient of the i-th node. Let be the computing power fluctuation parameter of the i-th node. Let be the stability weight parameter of the i-th node. This is the redundancy adjustment coefficient. Let i be the task priority parameter for the i-th node. This represents the total number of nodes.

3. The task allocation method for distributed data computing according to claim 1, characterized in that, The expression for the lightweight edge collaborative allocation model is: ,in, For task node matching degree, For the feature parameters of the j-th task, Let J be the feature weight parameters for the j-th task. Let be the collaborative capability parameter of the k-th edge node. Let be the communication weight parameter for the k-th edge node. Let j be the delay requirement parameter for the j-th task. Let J be the reliability weight parameter for the j-th task. Let be the distance parameter for the k-th edge node. Let be the bandwidth parameter of the k-th edge node. This represents the total number of tasks. This represents the total number of edge nodes. It is a cooperative operation operator.

4. The task allocation method for distributed data computing according to claim 1, characterized in that, The expression for the deep annealing fragmentation scheduling optimization network is: ,in, To optimize the target value for partitioned scheduling, Let t be the size parameter of the t-th data partition. Let be the importance weight parameter for the t-th data slice. This is the annealing coefficient. Let be the scheduling temperature change parameter for the t-th partition. Let be the scheduling adjustment parameters for the t-th partition. Let be the initial scheduling efficiency parameter for the t-th partition. Let be the target scheduling efficiency parameter for the t-th partition. This represents the total number of data shards.

5. The task allocation method for distributed data computing according to claim 1, characterized in that, The task status feedback model expression of the distributed task monitoring and computing platform is: ,in, This is the node status feedback value. These are the real-time operating parameters for the s-th node. These are the baseline operating parameters for the s-th node. Let be the feedback weight parameter of the s-th node. Let be the task completion progress parameter for the s-th node. Let be the anomaly warning weight parameter for the s-th node. The total number of monitoring indicators. For gradient operation operators, .

6. The task allocation method for distributed data computing according to claim 1, characterized in that, The expression for the task allocation efficiency model of distributed data computing is: ,in, Assign efficiency coefficients to tasks. The execution efficiency parameter for the z-th assigned task. The weight parameter for assigning the z-th task. Let be the efficiency fluctuation parameter for the z-th task. Let z be the transmission fluctuation parameter for the z-th task. Let be the collaborative gain parameter for the z-th task. Let be the resource consumption parameter for the z-th task. To allocate the total number of tasks.

7. The task allocation method for distributed data computing according to claim 1, characterized in that, S3 includes the following sub-steps: S31, extracting the computing power demand threshold, data processing type, task execution time constraints, and interaction dependencies from the task feature parameters to construct a task feature vector matrix; S32, quantizing and decomposing the computing power range corresponding to the node computing power level to generate the computing power supply vector of each node, including peak computing power, continuous computing power, and concurrent computing power calibration dimension data; S33, performing element-wise matching operations between the task feature vector matrix and the node computing power supply vector, calculating the cosine similarity and Euclidean distance between the vectors, and initially screening nodes with similarity higher than a set threshold and distance lower than a set threshold; S34, combining the node's historical task execution records to perform a secondary verification on the initially screened nodes, eliminating nodes with task conflicts and historical anomalies to form a candidate node set.

8. The task allocation method for distributed data computing according to claim 1, characterized in that, S4 includes the following sub-steps: S41, analyzing the total amount of data, data type, granularity requirements and integrity constraints in the data sharding parameters to determine the basic rules and boundary conditions for sharding; S42, based on the initial temperature parameters and cooling rate parameters of the deep annealing sharding scheduling optimization network, performing preliminary sharding processing on the data to generate several initial sharding units and recording the feature identifiers of each unit. S43 adjusts the initial sharding units through the network's iterative scheduling mechanism, merging excessively small sharding units and splitting excessively large sharding units, so that the size of each sharding unit is adapted to the computing power carrying capacity of the candidate nodes. S44. Based on the number of candidate nodes and the computing power ratio of each node, the adjusted sharding units are allocated to the corresponding nodes to form a sharding allocation scheme and mark the transmission order of each shard.

9. The task allocation method for distributed data computing according to claim 1, characterized in that, S5 includes the following sub-steps: S51, collecting bandwidth utilization, transmission delay, packet loss rate, and link stability data from network transmission parameters to construct a network transmission quality evaluation matrix; S52, performing a traversal analysis of the transmission paths between candidate nodes based on the evaluation matrix to select the optimal transmission path with the lowest transmission delay and packet loss rate; S53, formulating differentiated transmission timing plans based on the size and priority of each fragment in the fragmentation allocation scheme, prioritizing the transmission of high-priority fragments and fragments with smaller data volumes; S54, monitoring network status changes in real time during transmission, and dynamically adjusting the transmission path and timing plan when network parameters exceed a set threshold to ensure the continuity and stability of fragmented transmission.

10. A task allocation method for distributed data computing according to any one of claims 1-9, characterized in that, This method is implemented through different units, including: a multi-dimensional parameter acquisition and analysis unit, a computing power redundancy balancing and management unit, a lightweight edge collaboration matching unit, a deep annealing sharding scheduling optimization unit, a network transmission adaptive adjustment unit, and a task allocation final confirmation unit. The multi-dimensional parameter acquisition and analysis unit establishes a data interaction connection with the distributed task monitoring and computing platform to collect and analyze task characteristics, node computing power, network transmission, and data sharding correlation parameters. The computing power redundancy balancing and management unit receives the node computing power parameters output by the multi-dimensional parameter acquisition and analysis unit and divides the node computing power level and redundant computing power range. The lightweight edge collaboration matching unit connects with the multi-dimensional parameter acquisition and analysis unit. The unit and the computing power redundancy balance management unit are connected to each other to complete the candidate node selection based on task characteristic parameters and node computing power level; the deep annealing sharding scheduling optimization unit is connected to the multi-dimensional parameter acquisition and analysis unit to perform adaptive sharding processing on data sharding parameters and generate sharding allocation schemes; the network transmission adaptive adjustment unit receives the sharding allocation scheme output by the deep annealing sharding scheduling optimization unit and the network transmission parameters output by the multi-dimensional parameter acquisition and analysis unit, respectively, and optimizes the transmission path and timing; the task allocation final confirmation unit is connected to the distributed task monitoring computing platform and the network transmission adaptive adjustment unit to complete the final confirmation of the task allocation scheme based on node running status parameters.

Images