Data processing method and device based on distributed network and storage medium

Abstract

The invention relates to the technical field of distributed processing, and provides a data processing method based on a distributed network, and the method comprises the steps: collecting a dynamic feature set and a real-time resource index set of an input data flow in real time; inputting the dynamic feature set and the real-time resource index set into a pre-trained autonomous optimization model, and generating a joint optimized data fragmentation strategy matrix and a task allocation weight vector; splitting the to-be-processed task into a plurality of sub-tasks with an execution order relationship according to the data dependency graph, and dynamically mapping each sub-task to a specified storage fragment of the target node according to the weight vector; in the subtask execution process, each node continuously exchanges execution state data, and dynamically adjusts the task execution path weight of adjacent nodes according to the progress deviation rate and the resource margin in the state data so as to form a closed loop feedback optimization link; and according to the continuously updated real-time resource index set and the progress deviation rate, triggering a fragment migration transaction and a fault recovery assembly line, and carrying out strategy iteration updating.

Description

Technical Field

[0001] The present invention relates to the field of distributed processing technology, and in particular to a data processing method, device and storage medium based on a distributed network. Background Art

[0002] With the widespread adoption of cloud computing and edge computing, the scale of data processed by distributed systems is growing exponentially. Furthermore, the coupling between the dynamic characteristics of data streams (such as timing fluctuations and business relevance) and heterogeneous resource environments (such as fluctuations in node computing power and changes in network status) is becoming increasingly complex. Existing distributed data processing methods suffer from shortcomings such as the separation of data feature collection and resource monitoring, static policy-driven task allocation, a lack of dynamic coordination during execution, and the decoupling of fault recovery from policy iteration.

[0003] These technical deficiencies lead to existing methods experiencing low resource utilization, large task latency fluctuations, and long recovery cycles when handling highly dynamic and real-time tasks. Therefore, a distributed data processing approach is urgently needed that can achieve data-resource collaborative perception, dynamic policy optimization, and closed-loop elasticity. Summary of the Invention

[0004] The present application provides a data processing method, device and storage medium based on a distributed network, which can efficiently process massive data streams.

[0005] In one aspect, the present application provides a data processing method based on a distributed network, the method comprising: Collect dynamic feature sets of input data streams in real time and simultaneously obtain real-time resource indicator sets; Inputting the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and a task allocation weight vector; Based on the data sharding strategy matrix, the task to be processed is split into multiple subtasks with an execution order relationship according to the data dependency graph, and each subtask is dynamically mapped to a specified storage shard of the target node according to the weight vector; During the execution of subtasks, each node continuously exchanges execution status data and dynamically adjusts the task execution path weights of adjacent nodes based on the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; According to the continuously updated real-time resource indicator set and the progress deviation rate, the shard migration transaction and the fault recovery pipeline are triggered, and the event log is written back to the autonomous optimization model for strategy iteration and update.

[0006] On the other hand, the present application provides a data processing device based on a distributed network, the device comprising: The acquisition module is used to collect the dynamic feature set of the input data stream in real time and synchronously obtain the real-time resource indicator set; A generation module, configured to input the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and a task allocation weight vector; A splitting module is used to split the task to be processed into multiple subtasks with an execution order relationship according to the data dependency graph based on the data sharding strategy matrix, and dynamically map each subtask to a specified storage shard of the target node according to a weight vector; The adjustment module is used to continuously exchange execution status data between nodes during the execution of subtasks, and dynamically adjust the task execution path weights of adjacent nodes based on the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; The trigger module is used to trigger the shard migration transaction and fault recovery pipeline according to the continuously updated real-time resource indicator set and the progress deviation rate, and write back the event log to the autonomous optimization model for strategy iterative update.

[0007] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the technical solution of the above-mentioned distributed network-based data processing method are implemented.

[0008] In a fourth aspect, the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the technical solution of the above-mentioned distributed network-based data processing method.

[0009] From the technical solution provided by the above application, it can be seen that, on the one hand, the dynamic feature set of data and the real-time resource indicator set of nodes are collected synchronously to realize the multi-dimensional joint perception of data type distribution, timing pattern and CPU / storage / network status, and provide accurate real-time input for dynamic strategy generation, thereby significantly improving the matching degree between task allocation strategy and real-time resources; on the other hand, the joint optimization strategy matrix and weight vector are generated by the autonomous optimization model, and the sharding constraints and node priorities are dynamically adjusted based on the real-time data-resource coupling state, which can effectively reduce the task execution delay in heterogeneous resource scenarios and reduce resource fragmentation; thirdly, tasks are split according to the data dependency graph and dynamically mapped to the target shards, and the deployment location is constrained by the weight vector to ensure that subtask groups with strong dependencies are preferentially deployed to low-latency node clusters, thereby improving data access locality and task execution efficiency. In summary, the technical solution of the present application can efficiently process large stream data through a distributed network. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0012] Figure 1 is a flow chart of a data processing method based on a distributed network provided in an embodiment of the present application; Figure 2 is a structural diagram of a data processing device based on a distributed network provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0014] In this specification, adjectives such as first and second may be used only to distinguish one element or action from another element or action, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0015] In this specification, for the convenience of description, the sizes of various parts shown in the drawings are not drawn according to the actual proportions.

[0016] With the prevalence of cloud computing and edge computing, the scale of data that distributed systems need to process is growing exponentially, and the coupling relationship between the dynamic characteristics of data streams (such as timing fluctuations and business relevance) and heterogeneous resource environments (such as node computing power fluctuations and network status changes) is becoming increasingly complex. Existing distributed data processing methods suffer from the separation of data feature collection and resource monitoring, static policy-driven task allocation, lack of dynamic coordination during execution, and decoupling of fault recovery from policy iteration. The following technical approaches are commonly used: 1) Separation of data feature collection and resource monitoring: Data features and node resource indicators are collected through independent modules, resulting in the task allocation strategy being unable to perceive the dynamic correlation between data flow changes and resource status in real time; 2) Static policy-driven task allocation: Sharding strategies are generated based on offline trained models or predefined rules, which are difficult to adapt to dynamic adjustments in business topology and sudden resource bottlenecks. 3) Lack of dynamic coordination during execution: After task deployment, tasks rely on fixed execution paths and cannot be dynamically optimized based on real-time load. Furthermore, a global lock mechanism is used during shard migration, resulting in system response delays. 4) Fault recovery and policy iteration are decoupled: Migration and recovery operations do not form a closed-loop feedback loop, making similar faults prone to recurrence.

[0017] In view of the above problems of the prior art, this application proposes a data processing method based on a distributed network, the flow chart of which is shown in the attached figure. Figure 1 As shown, it mainly includes steps S101 to S105, which are detailed as follows: Step S101: Collect the dynamic feature set of the input data stream in real time, and synchronously obtain the real-time resource indicator set.

[0018] It's important to note that if only a single dimension is collected (e.g., only data features or only resource metrics), the subsequent autonomous optimization model won't be able to perceive the correlation between data features and resource fluctuations, causing sharding strategies and task allocation plans to deviate from the actual operating environment. For example, when data flows suddenly increase, node computing power can't be dynamically adapted. This means that by synchronously collecting a set of dynamic data features and a set of node resource metrics (such as data type distribution, timing patterns, and CPU / storage / network metrics), we can achieve joint modeling of data flows and resource states.

[0019] In an embodiment of the present application, the real-time collection of the dynamic feature set of the input data stream and the synchronous acquisition of the real-time resource indicator set are actually the multi-dimensional state perception stage of the data processing method based on the distributed network. Specifically, the dynamic feature set of the input data stream can be collected in real time through a distributed probe cluster deployed at the data source access layer, and the real-time resource indicator set can be obtained synchronously through the resource monitoring agent installed on the computing node, wherein the dynamic feature set includes the data type distribution, the data flow timing pattern and the business logic topology, and the real-time resource indicator set includes the CPU computing power fluctuation of each node, the dynamic occupancy rate of the storage space and the network bandwidth utilization rate. By synchronously collecting the dynamic feature set of data and the real-time resource indicator set of the node, the multi-dimensional joint perception of the data type distribution, timing pattern and CPU / storage / network status can be achieved, providing accurate real-time input for dynamic strategy generation, thereby significantly improving the matching degree between the task allocation strategy and the real-time resources. As an embodiment of the present application, the real-time collection of the dynamic feature set of the input data stream can be achieved through steps 1011 to S1013, as detailed below: Step 1011: A feature extractor is implanted in the data input pipeline to generate a data type distribution vector by identifying the proportion of structured and unstructured data.

[0020] Step 1012: Based on the data type distribution vector, extract the time series fluctuation characteristics of the data stream and encode them into a time series matrix.

[0021] The extraction of the timing fluctuation characteristics of the data stream can be achieved by performing wavelet transform analysis on the arrival interval of the data stream.

[0022] Step 1013: Input the time series matrix into the business rule engine to construct a dynamically updated business logic topology diagram.

[0023] Specifically, the time series matrix is input into the business rule engine, which parses the transaction tags carried by the data and combines them with the preset cross-source association rules to build a dynamically updated business logic topology map.

[0024] From step 1011 to step S1013 of the above embodiment, it can be seen that through hierarchical processing, namely data type identification-time series feature extraction-business topology construction, the complexity of feature extraction can be reduced, and data overload during business topology construction can be avoided, thereby improving the completeness and relevance of the dynamic feature set, so that the subsequent strategy generation model can accurately identify the mapping relationship between data flow and business logic.

[0025] Step S102: Input the dynamic feature set and the real-time resource indicator set into the pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and task allocation weight vector.

[0026] Considering that if a fixed sharding strategy (such as equal sharding) is adopted or tasks are allocated based solely on resource indicators, the shard size cannot be dynamically adjusted according to the business topology, resulting in wasted storage space or conflicting task execution paths. Therefore, this application inputs the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and task allocation weight vector. This can convert multi-dimensional perception data into executable sharding constraints and task priority rules, solving the adaptability defects of static strategies. The proposed strategy matrix contains constraints on the number, size, and storage location of shards, and the weight vector defines the processing priority of each node for different task types. In addition, when generating the data sharding strategy matrix, the following shard-node matching evaluation is performed: the strategy score of each potential sharding node combination is calculated: Score = α × (storage capacity margin / demand) + β × (inverse of network delay) + γ × (historical completion rate of similar tasks), where α, β, and γ are the weight coefficients of storage capacity margin / demand, inverse of network delay, and historical completion rate of similar tasks, respectively; candidate strategies with scores higher than the fitness threshold are retained to form the data sharding strategy matrix.

[0027] The training process of the autonomous optimization model of the above embodiment is implemented according to the following collaborative mechanism: a dual-input neural network is established, wherein the first input branch performs LSTM feature extraction on the time series encoding of the dynamic feature set, and the second input branch performs graph convolution processing on the spatial distribution encoding of the real-time resource indicator set. Then, the features extracted by the two input branches are fused to obtain time-space fusion features; the time-space fusion features are input into the strategy matrix generator and the weight vector predictor respectively, and the gradient reversal layer is used to optimize the two in adversarial training; based on the training results of the adversarial training, the weighted sum of the historical task execution delay and the resource waste rate is calculated as the joint loss function, and the fully connected layer parameters of the dual-input neural network are updated through back propagation.

[0028] In order to solve the policy lag problem caused by offline training of traditional models and optimize policy constraints in a targeted manner through feature learning and penalty mechanisms of migration events, the iterative update of the autonomous optimization model can be achieved through the following closed-loop feedback mechanism: the causes of shard migration triggers are classified and encoded into feature vectors, and added to the training dataset to optimize the generation logic of the policy matrix; for migration transactions caused by unreasonable shard constraints in the policy matrix, the migration frequency penalty coefficient is added to the loss function; incremental training is started during off-peak periods, and the hidden layer weight distribution of the neural network is adjusted using an updated dataset containing migration features, and an updated policy matrix and weight vector are generated.

[0029] Step S103: Based on the data sharding strategy matrix, split the task to be processed into multiple subtasks with an execution order relationship according to the data dependency graph, and dynamically map each subtask to a specified storage shard of the target node according to the weight vector.

[0030] Directly assigning unsplit tasks or randomly mapping subtasks, as in existing technologies, will lead to a surge in cross-node data transmission (for example, dependency-intensive tasks are dispersed to high-latency nodes), significantly increasing task completion time. Therefore, to ensure that the task execution path meets data locality requirements and that the weight vector constrains node selection priority, subtasks can be split according to the data dependency graph (DAG) and dynamically mapped to the target node. That is, based on the data sharding strategy matrix, the task to be processed is split into multiple subtasks with an execution order relationship according to the data dependency graph, and each subtask is dynamically mapped to the designated storage shard of the target node based on the weight vector. When dynamically mapping each subtask to the designated storage shard of the target node, dynamic binding can be implemented. That is, before the subtask is distributed, a resource reservation request is initiated to the target node. If the node's current load rate exceeds the processing capacity declared in its weight vector, a suboptimal node is reselected according to the sliding window mechanism, and a bidirectional pointer index is established between the subtask and the storage shard to ensure data locality during execution.

[0031] Specifically, as an embodiment of the present application, based on the data sharding strategy matrix, splitting the task to be processed into multiple subtasks with an execution order relationship according to the data dependency graph can be achieved through steps S1031 to S1033, as detailed below: Step S1031: Convert the task input into a data dependency graph, where nodes represent atomic computing operations and edges represent data transmission dependencies.

[0032] Specifically, by parsing task descriptions, analyzing computational operations and data volumes to assess complexity and density, and sorting out data transfer relationships to identify dependencies, a lightweight agent is used to collect node resource metrics such as CPU, memory, and network bandwidth in real time. Resource tags such as "GPU acceleration" and "high bandwidth" are then added to nodes based on the hardware and software environment. It should be noted that semantic analysis can be introduced when constructing the data dependency graph in the above embodiment. Specifically, the syntax tree of the SQL query can be parsed to identify JOIN operation dependency chains; pipeline dependencies for feature engineering can be analyzed for machine learning training tasks; and a data transmission cost weighting factor can be added to each dependency edge.

[0033] Step S1032: Perform critical path analysis on the data dependency graph to identify the subtask chain with the highest priority.

[0034] For example, task dependency graphs can be used to hierarchically break down complex, computationally intensive tasks into smaller pieces, while simpler tasks can be broken down into larger pieces. Combining task characteristics with node resource tags, tasks involving extensive matrix operations can be broken down into smaller pieces for GPU acceleration nodes, while tasks with extensive data transfer can be broken down into data transfer subtasks for high-bandwidth nodes.

[0035] Step S1033: Divide the data dependency graph into multiple subtask groups that meet the sharding constraints in the strategy matrix using a graph partitioning algorithm.

[0036] Specifically, subtasks can be prioritized based on task urgency and data dependencies, prioritizing high-priority subtasks. During allocation, nodes are traversed to find the node that best matches the subtask's resource requirements. For example, compute-intensive subtasks can be assigned to nodes with high CPU performance and low load. Simultaneously, task execution and node resources are monitored in real time, and subsequent subtasks are reassigned when node resources are limited. Task execution and resource utilization data are collected, and algorithms such as reinforcement learning are used to optimize task decomposition and allocation strategies.

[0037] From the above embodiments, it can be seen that by splitting tasks according to the data dependency graph and dynamically mapping them to the target shards, and combining the weight vector to constrain the deployment location, it can ensure that subtask groups with strong dependencies are preferentially deployed to low-latency node clusters, thereby improving data access locality and task execution efficiency.

[0038] Step S104: During the subtask execution process, each node continuously exchanges execution status data and dynamically adjusts the task execution path weights of adjacent nodes according to the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link.

[0039] Although the aforementioned data sharding strategy matrix is based on which the task to be processed is split into multiple subtasks with an execution order relationship according to the data dependency graph, and each subtask is dynamically mapped to the designated storage shard of the target node based on the weight vector, however, if only the initial deployment strategy is relied upon, the task path cannot be dynamically adjusted when a node suddenly fails or the load fluctuates, resulting in a system avalanche (for example, a backlog of tasks on a high-load node causes a chain delay). Therefore, it is necessary to dynamically adjust the task path weight based on state sharing and progress deviation rate to form a closed-loop feedback link and correct the execution deviation in real time. That is, during the execution of the subtask, each node continuously exchanges execution state data and dynamically adjusts the task execution path weight of the adjacent node based on the progress deviation rate and resource margin in the state data to form a closed-loop feedback optimization link. From the above embodiment, it can be seen that by dynamically adjusting the task execution path weight through state sharing and progress deviation rate feedback between nodes, a closed-loop optimization mechanism is formed, which can quickly balance the task queue when the node load suddenly changes and maintain the overall stability of the system.

[0040] Specifically, the closed-loop feedback optimization link formed in the above embodiment can be: each node publishes a status snapshot to the message bus every second, and the snapshot includes the current CPU remaining clock cycle, the depth of the queue to be processed, and the memory page error count; a regional coordinator is set to calculate the load balancing gradient based on the status snapshots of the nodes within a certain number of adjacent hops; when the absolute value of the load balancing gradient exceeds the adaptive threshold, a back pressure signal is injected into the downstream node of the high-load node to slow down the task injection rate. Among them, after the back pressure signal is injected into the downstream node of the high-load node, the following linkage operations are triggered: a checkpoint snapshot is generated for the subtask being processed by the back pressure node, and its data shard version number is recorded; based on the real-time resource saturation of the back pressure node, the priority weight of the shard to be migrated is calculated; based on the consistent hash ring, the replacement node that matches the priority weight is located and a migration transaction lock is established; the hot migration thread is started, and the shard copy marked as migratable is transferred to the replacement node through zero copy; after the migration is completed and the data verification is passed, the original shard space is released and the global routing table is updated. It should be noted that in the above-mentioned embodiment, the back pressure signal is a dynamic flow control mechanism. When a node (i.e., the node being back pressured) is unable to process tasks in a timely manner due to resource overload (e.g., CPU / queue overload), this signal is used to transmit the load status to the upstream node (i.e., the task sender), requesting it to reduce the task injection rate to avoid system avalanche. Accordingly, the node being back pressured refers to the computing node that triggers the back pressure signal due to resource overload. The establishment of a migration transaction lock based on the consistent hash ring positioning and priority weight matching of the replacement node can be achieved through steps S1041 to S1044, as detailed below: Step S1041: Construct a node hash ring.

[0041] Specifically, this involves node hash mapping and weight factor injection. Node hash mapping uses the IP addresses and port numbers of all available nodes in the cluster (including the current node and standby nodes) as input, generating a fixed-length hash value using a consistent hashing algorithm (such as SHA-256). This hash value is then mapped to a virtual ring space (0 to 2^128-1), with each node occupying one or more virtual positions on the ring based on the hash value. Weight factor injection involves calculating a priority weight (Weight = α × CPU headroom + β × memory headroom + γ × bandwidth headroom, where α + β + γ = 1) based on the node's real-time resource status (CPU remaining computing power, memory availability, and network bandwidth headroom). This weight is then proportionally converted to the number of virtual nodes: nodes with higher weights are allocated more virtual nodes, increasing their probability of being selected in the hash ring.

[0042] Step S1042: Locate and screen candidate nodes.

[0043] Specifically, locating and screening candidate nodes may include querying the hash ring, matching priority weights, and handling exceptions, among others. The implementation process of querying the hash ring includes: inputting the unique identifier of the shard to be migrated (e.g., shard ID + version number), calculating its hash value using the same hash algorithm, and searching for the nearest virtual node clockwise on the ring; locating the physical node corresponding to the first virtual node as the initial candidate node. The implementation process of matching priority weights includes: obtaining the real-time priority weight value of the candidate node. If the weight value is ≥ a preset threshold (e.g., weight ≥ 0.7), it is directly selected as the replacement node; if the weight does not meet the standard, continue to search for the next N physical nodes (N is configurable) clockwise along the ring, and select the node with the highest weight as the replacement node. Handling exceptions mainly includes triggering the exception marking mechanism if the candidate node is offline or has insufficient resources, temporarily removing the node from the hash ring, and re-executing the above-mentioned querying the hash ring and matching priority weights.

[0044] Step S1043: Create a migration transaction lock.

[0045] Specifically, establishing a migration transaction lock involves applying for a transaction lock, handling transaction lock conflicts, and holding and resuming the transaction lock. The process for applying for a transaction lock can be as follows: a migration transaction lock request is initiated to a global transaction coordinator (e.g., a distributed lock service based on the Raft protocol). The request includes metadata such as the ID and version number of the shard to be migrated, the source and target node IDs, and the migration priority weight. The coordinator then verifies the status of the target node. If migration is possible, the lock is granted and the transaction ID is returned; otherwise, a rejection signal is returned. The process for handling transaction lock conflicts can be as follows: if another migration transaction is detected to have locked the same shard or target node, a decision is made based on priority weight: the higher-weighted transaction is granted the lock first, while the lower-weighted transaction is placed in a waiting queue. If the weights are the same, transactions are sorted by their initiation timestamps. Holding and resuming a transaction lock primarily involves the coordinator automatically renewing the lock and triggering an alarm if the migration times out. If consecutive renewal attempts fail, the lock is forcibly released and the transaction is rolled back.

[0046] Step S1044: Commit the migration transaction and release the transaction lock.

[0047] Migration transaction submission is primarily atomic. After the migration is complete, a commit request is sent to the coordinator, along with a data checksum (e.g., a CRC64 checksum). The coordinator then verifies the checksum consistency and atomically updates the global routing table, binding the shard to the new node. Releasing transaction locks involves: upon successful submission, the coordinator releases the transaction lock and clears temporary metadata. If the submission fails, a rollback process is triggered: the target node's data replica is deleted, the source node's shard is restored to a writable state, and the transaction lock is released.

[0048] From step S1041 to step S1044 of the above embodiment, it can be seen that, on the one hand, the distribution of virtual nodes is dynamically adjusted through weights to ensure that nodes with high resource margins are more easily selected, thereby significantly improving the load balance of the cluster after migration; on the other hand, the transaction lock mechanism can significantly reduce the shard migration conflict rate; in addition, atomic commit based on version number and check code can reduce the probability of data consistency deviation during migration to a preset threshold, for example, below 0.01%.

[0049] Step S105: Based on the continuously updated real-time resource indicator set and progress deviation rate, the shard migration transaction and fault recovery pipeline are triggered, and the event log is written back to the autonomous optimization model for strategy iteration and update.

[0050] If, as in the prior art, only migration recovery is performed without feedback to the model, similar faults will be repeatedly triggered (e.g., the same resource bottleneck recurs), and system reliability will gradually deteriorate. The embodiments of the present application can trigger shard migration and fault recovery, and write back event logs to the model, converting runtime anomalies into policy iteration basis, thereby achieving a closed loop of fault self-healing and policy optimization. Specifically, based on the continuously updated real-time resource indicator set and progress deviation rate, when it is detected that the node resource saturation exceeds the dynamic threshold or the shard data consistency deviates from the preset tolerance, the shard migration transaction and fault recovery pipeline are triggered, and the event log is written back to the autonomous optimization model for policy iteration update.

[0051] The shard migration transaction of the above embodiment can implement hierarchical atomic operations, that is, all replica paths of the shard to be migrated are locked in the global naming service, and a migration transaction ID containing the path fingerprint is generated; based on the locked replica paths, a cross-node Remote Direct Memory Access (RDMA) channel is established to batch migrate data blocks, and the source shard is switched to read-only mode; after all target nodes verify the integrity of the data blocks through the replica paths, the shard location mapping table is atomically updated and the migration transaction ID is released. During the execution of the shard migration transaction, a request redirection agent can also be implemented for the shard access request issued by the client to guide it to the latest available replica; for read and write operation conflicts generated during the redirection process, multi-version concurrency control (MVCC) is used to generate a data version branch with a timestamp; after the transaction is completed, the data version branch is merged into a linearized sequence according to the timestamp priority and business consistency rules; the invalid expired data copies after the merge are cleared, and the associated storage space is released. As for triggering the fault recovery pipeline, specifically, when the node temperature sensor detects continuous overtemperature, the frequency of the central processing unit is automatically reduced and the backup cooling unit is started; if the memory error correction code error rate exceeds the chip specification threshold, the memory stick with the error correction code error rate exceeding the chip specification threshold is isolated and switched to the mirror channel; for unrepairable hardware failures, the standby node cold start process is triggered and the pre-written log recovery status is replayed.

[0052] To reduce storage overhead and ensure data integrity and service continuity after the failed node is restored, the cold start process includes the following intelligent state reconstruction: restoring the metadata index from the latest global checkpoint, where the index contains the shard topology relationship and storage location mapping table; locating the erasure code distribution node of the missing data shard based on the storage location mapping table, and reconstructing the complete data shard through a decoding algorithm; resubscribing to the original topic partition of the message bus based on the message subscription relationship configuration in the metadata index, and resuming the data stream based on the checkpoint timestamp.

[0053] The above embodiment may also include a collaborative comparison mechanism across data centers, that is, periodically generating a data fingerprint matrix and performing cross-center comparison through blockchain evidence storage; when the fingerprint difference exceeds a security threshold, locating the abnormal shard and triggering consistency repair; implementing trust downgrade for maliciously tampered nodes and isolating them to a sandbox environment.

[0054] From the above attached Figure 1From the example of the data processing method based on the distributed network, it can be seen that, on the one hand, the dynamic feature set of data and the real-time resource indicator set of the node are collected synchronously to realize the multi-dimensional joint perception of data type distribution, timing pattern and CPU / storage / network status, and provide accurate real-time input for dynamic strategy generation, thereby significantly improving the matching degree between task allocation strategy and real-time resources; on the other hand, the joint optimization strategy matrix and weight vector are generated by the autonomous optimization model, and the sharding constraints and node priorities are dynamically adjusted based on the real-time data-resource coupling state, which can effectively reduce the task execution delay in heterogeneous resource scenarios and reduce resource fragmentation; thirdly, tasks are split according to the data dependency graph and dynamically mapped to the target shards, and the deployment location is constrained by the weight vector to ensure that subtask groups with strong dependencies are preferentially deployed to low-latency node clusters, thereby improving data access locality and task execution efficiency. In summary, the technical solution of the present application can efficiently process large stream data through a distributed network.

[0055] Please see the attached Figure 2 , is a data processing device based on a distributed network provided in an embodiment of the present application. The device may include an acquisition module 201, a generation module 202, a splitting module 203, an adjustment module 204, and a triggering module 205, which are described in detail as follows: Acquisition module 201, used to collect dynamic feature sets of input data streams in real time and synchronously acquire real-time resource indicator sets; A generation module 202 is configured to input the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and a task allocation weight vector; Splitting module 203 is used to split the task to be processed into multiple subtasks with execution order relationships according to the data dependency graph based on the data sharding strategy matrix, and dynamically map each subtask to a specified storage shard of the target node according to the weight vector; Adjustment module 204 is used to continuously exchange execution status data between nodes during subtask execution, and dynamically adjust the task execution path weights of adjacent nodes based on the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; The trigger module 205 is used to trigger the shard migration transaction and fault recovery pipeline according to the continuously updated real-time resource indicator set and progress deviation rate, and write back the event log to the autonomous optimization model for strategy iteration update.

[0056] From the above attached Figure 2It can be seen from the example of a data processing device based on a distributed network that, on the one hand, it synchronously collects data dynamic feature sets and node real-time resource indicator sets to achieve multi-dimensional joint perception of data type distribution, timing patterns and CPU / storage / network status, and provides accurate real-time input for dynamic strategy generation, thereby significantly improving the matching degree between task allocation strategies and real-time resources; on the other hand, it generates a joint optimization strategy matrix and weight vector through an autonomous optimization model, and dynamically adjusts shard constraints and node priorities based on real-time data-resource coupling status, which can effectively reduce task execution delays in heterogeneous resource scenarios and reduce resource fragmentation; thirdly, it splits tasks according to the data dependency graph and dynamically maps them to target shards, and combines the weight vector to constrain the deployment location to ensure that subtask groups with strong dependencies are preferentially deployed to low-latency node clusters, thereby improving data access locality and task execution efficiency. In summary, the technical solution of the present application can efficiently process large stream data through a distributed network.

[0057] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a data processing method based on a distributed network. When the processor 30 executes the computer program 32, the steps in the embodiment of the data processing method based on a distributed network are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned device embodiments are realized, for example Figure 2 The functions of the acquisition module 201, the generation module 202, the splitting module 203, the adjustment module 204 and the triggering module 205 are shown.

[0058] Exemplarily, a computer program 32 of a distributed network-based data processing method mainly includes: real-time acquisition of a dynamic feature set of an input data stream and synchronous acquisition of a real-time resource indicator set; inputting the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and a task allocation weight vector; based on the data sharding strategy matrix, splitting the task to be processed into multiple subtasks with an execution order relationship according to the data dependency graph, and dynamically mapping each subtask to a designated storage shard of the target node according to the weight vector; during the execution of the subtasks, each node continuously exchanges execution status data and dynamically adjusts the task execution path weights of adjacent nodes based on the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; based on the continuously updated real-time resource indicator set and progress deviation rate, triggering shard migration transactions and fault recovery pipelines, and writing back event logs to the autonomous optimization model for strategy iterative update. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 may be divided into the functions of an acquisition module 201, a generation module 202, a splitting module 203, an adjustment module 204, and a triggering module 205 (modules in the virtual device), and the specific functions of each module are as follows: the acquisition module 201 is used to collect the dynamic feature set of the input data stream in real time, and synchronously obtain the real-time resource indicator set; the generation module 202 is used to input the dynamic feature set and the real-time resource indicator set into the pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and task allocation weight vector; the splitting module 203 is used to allocate the tasks to be processed based on the data sharding strategy matrix. The task is split into multiple subtasks with an execution order relationship according to the data dependency graph, and each subtask is dynamically mapped to the specified storage shard of the target node based on the weight vector; the adjustment module 204 is used to continuously exchange execution status data among the nodes during the execution of the subtasks, and dynamically adjust the task execution path weights of adjacent nodes according to the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; the trigger module 205 is used to trigger the shard migration transaction and fault recovery pipeline according to the continuously updated real-time resource indicator set and progress deviation rate, and write back the event log to the autonomous optimization model for strategy iterative update.

[0059] The electronic device 3 may include but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3It is only an example of electronic device 3 and does not constitute a limitation of electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0060] The processor 30 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0061] The memory 31 can be an internal storage unit of the electronic device 3, such as the hard drive or memory of the electronic device 3. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 31 can include both the internal storage unit of the electronic device 3 and an external storage device. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or is about to be output.

[0062] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0063] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0064] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0065] In the embodiments provided in this application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0066] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0067] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0068] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing related hardware through a computer program. The computer program of the data processing method based on a distributed network can be stored in a storage medium. When the computer program is executed by the processor, it can implement the steps of each of the above-mentioned method embodiments, namely, real-time acquisition of the dynamic feature set of the input data stream and synchronous acquisition of the real-time resource indicator set; inputting the dynamic feature set and the real-time resource indicator set into the pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and task allocation weight vector; based on the data sharding strategy matrix, splitting the task to be processed into multiple subtasks with an execution order relationship according to the data dependency graph, and dynamically mapping each subtask to the designated storage shard of the target node according to the weight vector; during the execution of the subtasks, each node continuously exchanges execution status data and dynamically adjusts the task execution path weights of adjacent nodes according to the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; based on the continuously updated real-time resource indicator set and progress deviation rate, triggering the shard migration transaction and fault recovery pipeline, and writing back the event log to the autonomous optimization model for policy iterative update. Computer programs include computer program code, which may be in source code, object code, executable files, or some intermediate form. Storage media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunications signals, and software distribution media. It should be noted that the content of storage media may be appropriately expanded or reduced based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, storage media do not include electric carrier signals or telecommunications signals.

[0069] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application. The specific implementation methods described above further explain the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present invention.

Claims

1. A data processing method based on a distributed network, characterized in that: The method comprises: Collect dynamic feature sets of input data streams in real time and simultaneously obtain real-time resource indicator sets; Inputting the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and a task allocation weight vector; Based on the data sharding strategy matrix, the task to be processed is split into multiple subtasks with an execution order relationship according to the data dependency graph, and each subtask is dynamically mapped to a specified storage shard of the target node according to the weight vector; During the execution of subtasks, each node continuously exchanges execution status data and dynamically adjusts the task execution path weights of adjacent nodes based on the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; According to the continuously updated real-time resource indicator set and the progress deviation rate, the shard migration transaction and the fault recovery pipeline are triggered, and the event log is written back to the autonomous optimization model for strategy iteration and update.

2. The data processing method based on a distributed network according to claim 1, characterized in that: The dynamic feature set of the real-time acquisition input data stream includes: A feature extractor is embedded in the data input pipeline to generate a data type distribution vector by identifying the proportion of structured and unstructured data. Based on the data type distribution vector, extract the time series fluctuation characteristics of the data stream and encode them into a time series matrix; The time series matrix is input into a business rule engine to construct a dynamically updated business logic topology diagram.

3. The data processing method based on a distributed network according to claim 1, characterized in that: The forming of a closed-loop feedback optimization link includes: Each node publishes a status snapshot to the message bus every second. The snapshot includes the current remaining CPU clock cycles, the depth of the pending queue, and the memory page fault count. Set up a regional coordinator to calculate the load balancing gradient based on the state snapshots of nodes within a certain number of hops; When the absolute value of the load balancing gradient exceeds the adaptive threshold, a back pressure signal is injected into the downstream node of the high-load node to slow down the task injection rate.

4. The data processing method based on a distributed network according to claim 3, characterized in that: The injection of a back pressure signal to the downstream node of the high-load node triggers the following linkage operations: Generate a checkpoint snapshot for the subtask being processed by the back-pressured node and record its data shard version number; Calculate the priority weight of the shard to be migrated based on the real-time resource saturation of the back-pressured node; Based on the consistent hash ring, the replacement node that matches the priority weight is located and a migration transaction lock is established; Start the hot migration thread and transfer the shard copies marked as migratable to the replacement node through zero copy; After the migration is complete and the data verification is passed, the original shard space is released and the global routing table is updated.

5. The data processing method based on a distributed network according to claim 1, characterized in that: The triggering shard migration transaction failure recovery pipeline includes: When the node temperature sensor detects a continuous over-temperature, it automatically reduces the CPU frequency and activates the backup cooling unit; If the memory ECC error rate exceeds the chip specification threshold, the memory module with the ECC error rate exceeding the chip specification threshold is isolated and switched to the mirror channel; For irreparable hardware failures, the standby node cold start process is triggered and the write-ahead log is replayed to recover the status.

6. The data processing method based on a distributed network according to claim 5, characterized in that: The cold start process execution includes the following intelligent state reconstruction: Restore the metadata index from the latest global checkpoint, where the index includes the shard topology and storage location mapping table; Based on the storage location mapping table, locate the erasure code distribution node of the missing data shard, and reconstruct the complete data shard through a decoding algorithm; According to the message subscription relationship configuration in the metadata index, resubscribe to the original topic partition of the message bus and resume the data stream based on the checkpoint timestamp.

7. The data processing method based on a distributed network according to claim 1, characterized in that: The iterative update of the autonomous optimization model is achieved through the following closed-loop feedback mechanism: The shard migration triggering reasons are categorized and encoded into feature vectors, which are then added to the training dataset to optimize the generation logic of the policy matrix. For migration transactions caused by unreasonable sharding constraints in the policy matrix, a migration frequency penalty coefficient is added to the loss function; Incremental training is initiated during off-peak periods, and the updated dataset containing the migration features is used to adjust the hidden layer weight distribution of the neural network and generate an updated policy matrix and weight vector.

8. A data processing device based on a distributed network, characterized in that: The device comprises: The acquisition module is used to collect the dynamic feature set of the input data stream in real time and synchronously obtain the real-time resource indicator set; A generation module, configured to input the dynamic feature set and the real-time resource indicator set into a pre-trained autonomous optimization model to generate a jointly optimized data sharding strategy matrix and a task allocation weight vector; A splitting module is used to split the task to be processed into multiple subtasks with an execution order relationship according to the data dependency graph based on the data sharding strategy matrix, and dynamically map each subtask to a specified storage shard of the target node according to a weight vector; The adjustment module is used to continuously exchange execution status data between nodes during the execution of subtasks, and dynamically adjust the task execution path weights of adjacent nodes based on the progress deviation rate and resource margin in the status data to form a closed-loop feedback optimization link; The trigger module is used to trigger the shard migration transaction and fault recovery pipeline according to the continuously updated real-time resource indicator set and the progress deviation rate, and write back the event log to the autonomous optimization model for strategy iterative update.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images