Low-delay data co-processing method, device and equipment and storage medium

Abstract

The invention relates to the technical field of data processing, can be applied to the fields of medical health, financial science and technology and the like, and discloses a low-delay data co-processing method, device and equipment and a storage medium, and the method comprises the steps: configuring a data agent at a plurality of edge nodes, and accessing a local data source; in response to the query task, analyzing the query task into a plurality of sub-tasks through the semantic metadata mapping model; state information of the edge nodes is obtained, the sub-tasks are distributed to all the edge nodes through a delay sensing scheduling algorithm, and a task distribution scheme is generated; according to the task allocation scheme and a local data source, executing the corresponding sub-task at each edge node to generate a local result; and integrating the local results at the cloud to generate a global analysis result. According to the method, the data proxy is configured at the edge node, and the local data source is accessed, so that data in-situ analysis is realized, a delay sensing scheduling algorithm is matched, time consumption of data transmission and centralized processing is reduced, and the problem of high delay is solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology and can be applied to fields such as healthcare and financial technology. In particular, it relates to a low-latency data collaborative processing method, apparatus, device, and storage medium. Background Technology

[0002] In existing technologies, enterprise-level data analysis systems generally rely on ETL (Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from a source to a destination) as their core data processing paradigm. This model extracts raw data from various source systems, transmits it to a centralized processing center for cleaning, transformation, and integration, and finally loads it into a data warehouse or data lake to support subsequent batch analysis and report generation.

[0003] However, with the deepening development of scenarios such as the Internet of Things and real-time intelligent decision-making, especially in the healthcare and fintech fields with stringent requirements for timeliness, security, and heterogeneity, the inherent defects of traditional ETL architectures have become increasingly prominent, becoming a key bottleneck restricting business innovation and efficiency improvement. Specifically, this manifests in the following ways: 1. In healthcare settings, applications such as intensive care and remote patient monitoring rely on real-time or near real-time analysis of vital signs data (such as ECG and blood oxygen saturation) to trigger immediate alerts or clinical interventions. In the fintech field, high-frequency trading, real-time fraud prevention, and credit risk monitoring also require millisecond-level processing capabilities. In traditional ETL processes, data must be transmitted from edge devices (such as monitors, ATMs, and trading terminals) via network to a cloud data center for processing. This long and complex process results in significant delays in analysis, making it impossible to support the aforementioned critical business operations with extremely high timeliness requirements.

[0004] 2. Traditional architectures highly concentrate computing load in the cloud, failing to effectively utilize the computing power of edge devices. For example, in medical scenarios, various intelligent monitoring devices and mobile medical terminals possess certain computing capabilities, yet they are only used for data collection and uploading; in the financial sector, branch terminals and smart teller machines also have a large amount of idle computing power. This model of central overload and edge idleness not only drives up the construction and operation costs of central data centers but also limits the overall throughput and scalability of the system.

[0005] 3. Medical and health data and financial data are both highly sensitive personal information and are subject to strict domestic and international regulations. Traditional ETL requires the full transmission and storage of large amounts of raw sensitive data at a central node, significantly increasing the exposure of data during transmission and storage, increasing the risks of data leakage, tampering, and violations, and making compliance costs high and complex.

[0006] 4. In the medical environment, data comes from imaging equipment, laboratory systems, wearable sensors, medical records, etc., with vastly different formats and protocols. Similarly, the financial sector faces the challenge of integrating multiple types of data, including structured transaction data, unstructured customer feedback, and streaming market data. Traditional ETL relies on customized, fixed transformation rules and pipelines for each type of data source, resulting in rigid processes, long development cycles, and difficulty in quickly adapting to new data sources or changing analytical needs. This severely hinders the implementation of innovative applications based on multimodal data fusion, such as precision medicine and personalized financial services.

[0007] In conclusion, driven by the urgent need for digital and intelligent transformation in industries such as healthcare and fintech, the traditional ETL data processing model, which is based on centralized and batch processing, faces severe challenges in terms of real-time performance, resource efficiency, data security, and processing agility. Summary of the Invention

[0008] The purpose of this invention is to provide a low-latency data collaborative processing method, apparatus, device, and storage medium, which aims to solve the problems of high latency and low real-time performance in traditional ETL processes.

[0009] In a first aspect, embodiments of the present invention provide a low-latency data collaborative processing method, comprising: Configure data proxies on multiple edge nodes to connect to local data sources; In response to a query task, the query task is parsed into multiple subtasks using a semantic metadata mapping model; Obtain the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; Based on the task allocation scheme and the local data source, the corresponding sub-tasks are executed at each edge node to generate local results; The local results are integrated in the cloud to generate global analysis results.

[0010] Secondly, embodiments of the present invention provide a low-latency data collaborative processing apparatus, comprising: The configuration unit is used to configure data proxies across multiple edge nodes and connect to local data sources; The parsing unit is used to parse the query task into multiple subtasks in response to the query task through a semantic metadata mapping model; The allocation unit is used to acquire the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth. An execution unit is configured to execute the corresponding subtasks at each edge node according to the task allocation scheme and the local data source, and generate local results; The integration unit is used to integrate the local results in the cloud to generate global analysis results.

[0011] Thirdly, embodiments of the present invention provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the low-latency data collaborative processing method described in the first aspect.

[0012] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements the low-latency data collaborative processing method described in the first aspect.

[0013] This invention discloses a low-latency data collaborative processing method, apparatus, device, and storage medium, comprising: configuring data proxies at multiple edge nodes and accessing local data sources; responding to a query task, parsing the query task into multiple sub-tasks through a semantic metadata mapping model; acquiring the status information of the edge nodes, and using a latency-aware scheduling algorithm to allocate the sub-tasks to each edge node, generating a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; executing the corresponding sub-tasks at each edge node according to the task allocation scheme and the local data source, generating local results; and integrating the local results in the cloud to generate global analysis results. This invention achieves in-situ data analysis by configuring data proxies at edge nodes and accessing local data sources. Simultaneously, the latency-aware scheduling algorithm reduces data transmission and centralized processing time, solving the high latency problem. Secondly, the task allocation scheme coordinates edge and central resources, avoiding resource idleness and overload. Furthermore, relying on the semantic metadata mapping model, cross-source logical queries are implemented, reducing the risk of privacy leakage and solving the problem of heterogeneous data integration. This significantly reduces data processing latency, improves resource utilization, and enhances data security and compliance. The present invention also provides a low-latency data collaborative processing device, a computer-readable storage medium, and a computer device, which have the above-mentioned beneficial effects, and will not be described in detail here. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of an application environment for a low-latency data collaborative processing method according to an embodiment of the present invention; Figure 2 A flowchart illustrating a low-latency data collaborative processing method; Figure 3 A schematic block diagram of a low-latency data collaborative processing device; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more of its features, integrals, steps, operations, elements, components and / or collections thereof.

[0018] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0019] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] The low-latency data collaborative processing method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can configure data proxies on multiple edge nodes through the client to access local data sources; in response to query tasks, it parses the query task into multiple sub-tasks using a semantic metadata mapping model; it obtains the status information of the edge nodes, uses a latency-aware scheduling algorithm to allocate the sub-tasks to each edge node, and generates a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; according to the task allocation scheme and the local data source, it executes the corresponding sub-tasks on each edge node to generate local results; and it integrates the local results in the cloud to generate global analysis results. In this invention, by configuring data proxies on edge nodes to access local data sources, in-situ data analysis is achieved. Simultaneously, the latency-aware scheduling algorithm reduces data transmission and centralized processing time, solving the high latency problem; secondly, the task allocation scheme coordinates edge and central resources, avoiding resource idleness and overload; and thirdly, cross-source logical queries are implemented based on the semantic metadata mapping model, reducing the risk of privacy leakage and solving the problem of heterogeneous data integration. This significantly reduces data processing latency, improves resource utilization, and enhances data security and compliance. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, AR devices, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0021] Please see Figure 2 and Figure 3 This embodiment provides a low-latency data collaborative processing method, including: S101: Configure data proxies on multiple edge nodes to access local data sources; In this embodiment, configuring data proxies on multiple edge nodes and accessing local data sources includes: Configure lightweight data proxies for multiple edge nodes and determine the local data source type for each edge node; Based on the lightweight data proxy configured on each edge node, it connects to the corresponding type of local data source, which includes sensor streams, business databases, and log systems. The lightweight data proxy at each edge node calls the lightweight data virtualization engine to perform unified access adaptation to the connected local data source and generate data source access adaptation results. At each edge node, based on the data source access adaptation results, data summarization and feature extraction are performed on the local data source to generate a data feature summary. It monitors data change events from the connected local data source in real time, and outputs data change correlation information by combining data feature summaries.

[0022] This embodiment, by deploying a lightweight data agent and integrating a virtualization engine at each edge node, can adapt to various heterogeneous data sources such as sensor streams and databases through a unified interface, effectively solving the problem of difficult integration of multi-source heterogeneous data and providing a flexible and standardized data access foundation for subsequent processing. Secondly, by performing data summarization and feature extraction locally at each edge node, only high-value feature data is generated and uploaded, significantly reducing the amount of data that needs to be transmitted to the cloud. This directly reduces network bandwidth pressure and transmission latency. Simultaneously, since the original sensitive data does not need to leave the local machine, the risk of data leakage during transmission is greatly reduced, enhancing privacy protection and compliance. Finally, by monitoring local data change events in real time through the agent layer and associating them with the extracted data feature summaries, the system can instantly perceive changes in data status. This enables real-time monitoring, immediate alerts, and dynamic decision-making, effectively overcoming the high latency defects of traditional batch processing ETL modes and supporting scenarios with extremely high timeliness requirements, such as real-time device status monitoring and instant process optimization.

[0023] Specifically, lightweight data brokers are deployed across multiple edge nodes. First, adaptable data brokers are configured for each edge node based on its hardware capabilities and network environment, identifying and registering the types of local data sources that the nodes can connect to, including different formats such as structured databases, real-time data streams, and text logs.

[0024] After configuration, the data agent program on each node connects to the corresponding type of local data source through a standardized interface. Specifically, the agent program establishes a continuous connection channel with the sensor data stream, configures read permissions and access paths for the business database, and sets the monitoring directory and parsing rules for the log system, enabling comprehensive access to multiple types of data sources.

[0025] Subsequently, the data proxy program on each edge node activates the data virtualization function to abstract the various local data sources that have been connected. This function transforms data sources with different formats, protocols, and structures into a unified logical view, shielding underlying differences and generating standardized data source access adaptation results, providing a consistent access experience for upper-layer applications.

[0026] Building upon this, edge nodes utilize the data source access adaptation results to perform lightweight computations on local data. This computation process extracts core features of the data, such as statistical distribution, key indicators, and pattern characteristics, generating a data feature summary significantly smaller than the original data, thus reducing the burden of subsequent transmission.

[0027] Finally, the data broker continuously monitors changes to the local data source. When a new, modified, or deleted event is detected, it intelligently determines the relevance and importance of the change by combining it with a pre-generated data feature summary. This generates data change correlation information that includes the content of the change, its scope of impact, and related dimensions, providing a real-time data foundation for subsequent distributed query processing.

[0028] In the healthcare field, lightweight data proxy programs are configured at multiple edge nodes within hospitals, such as inpatient ward nursing stations, operating room edge servers, and mobile emergency units. Based on the scenario of each node, the corresponding local data source type is determined, including sensor data streams continuously monitoring patient vital signs, business databases storing electronic medical records, and medical equipment operation log systems.

[0029] Then, based on the configured lightweight data proxy, each node connects to the corresponding type of local data source. The nursing station proxy connects to the real-time vital signs stream generated by the multi-parameter monitor and the ward medical record database; the operating room proxy connects to the anesthesia machine, surgical equipment sensors, and surgical information database; and the emergency unit proxy connects to the portable monitor data stream and the vehicle terminal log.

[0030] Next, the lightweight data agent of each node calls the embedded lightweight data virtualization function module to uniformly access and adapt data sources with different protocols and formats. For example, it uniformly converts HL7 medical information standard data and device private protocol stream data into an internally processable time-series data model to generate data source access adaptation results.

[0031] Then, at each edge node, based on the access adaptation results, the local data source is processed in real time. For example, summaries of indicators such as heart rate variability and blood oxygen saturation trend are calculated in real time from the vital signs stream; key diagnostic and medication features are extracted from medical record texts to generate standardized data feature summaries.

[0032] Next, a lightweight data agent monitors events from the connected data sources in real time, such as monitor alarms, new medical record writes, or device malfunction logs. When an event occurs, it is immediately analyzed in conjunction with the latest data feature summary, outputting data change correlation information including the context of abnormal signs, medical record update snapshots, and device status, for use by the clinical real-time decision support system.

[0033] In the fintech field, lightweight data proxy programs are configured at multiple edge nodes of financial institutions, such as branch servers in various locations, ATM cluster control terminals, and mobile business processing terminals. The types of local data sources corresponding to each node are determined, including real-time transaction logs, local customer information databases, terminal operation log systems, and market data push streams.

[0034] Based on the configuration, lightweight data proxies at each edge node connect to the corresponding data sources. Branch server proxies connect to the counter transaction system database and the local risk control list database; ATM terminal proxies connect to deposit and withdrawal transaction streams and equipment operation logs; mobile terminal proxies connect to APP transaction request streams and customer behavior logs.

[0035] Then, through each agent, the embedded lightweight data virtualization function module is called to perform unified access adaptation for structured transaction data, unstructured log text, and streaming market data, converting all data into a unified financial event model within the framework and generating data source access adaptation results.

[0036] Next, at each edge node, local data is processed in real time based on this result. For example, suspicious transaction patterns (such as high frequency, large amount, and unusual timing) are statistically analyzed in real time from transaction logs; session features are extracted from customer behavior logs; and data feature summaries containing risk indicators, behavioral profiles, and compliance summaries are generated.

[0037] Subsequently, a lightweight data agent continuously monitors data changes, such as large transaction triggers, blacklist matching, or abnormal system access logs. Once such an event is detected, it is immediately correlated with the current data feature summary for review and analysis, outputting data change correlation information including transaction risk scores, related customer profiles, and abnormal operation links, for the real-time anti-fraud and risk control system to make instantaneous decisions.

[0038] S102: In response to the query task, the query task is parsed into multiple sub-tasks through the semantic metadata mapping model; In this embodiment, the query task is parsed into multiple subtasks through a semantic metadata mapping model, including: Receive the user's original query request and generate a structured query statement; Input the structured query representation into the semantic metadata mapping model to generate cross-source data logical mapping relationships; Identify the topology of the data sources involved in the query based on the cross-source data logical mapping relationship, and generate a data source dependency graph; Query fragments are identified based on the data source dependency graph, and a set of query fragments is generated. Perform semantic integrity verification on the set of query fragments and generate query units that pass the verification; Determine the execution order constraints between query units and generate an execution dependency matrix; The computational complexity of each query unit is calculated, and the computational complexity is combined with the execution dependency matrix to generate multiple subtasks and their corresponding scheduling priority sequences.

[0039] This embodiment utilizes a semantic model to generate cross-source data logical mapping relationships, enabling the system to intelligently understand the semantic associations between distributed data sources and support efficient cross-source queries. By identifying the data source topology and generating a dependency graph, the system achieves modular decomposition of queries, forming query fragments that can be processed in parallel, significantly improving processing efficiency. Simultaneously, semantic integrity verification ensures the accuracy and reliability of query units, preventing error propagation. Furthermore, by combining the execution dependency matrix and computational complexity, the system can intelligently determine the scheduling priority of subtasks, optimize the execution order, and reduce overall latency, making it particularly suitable for distributed environments and enhancing system scalability and resource utilization efficiency.

[0040] Specifically, the system receives raw query requests from users via an interactive interface or API, which may be in the form of natural language or SQL-like languages. The system first performs syntax parsing and intent understanding on the request, transforming it into an internally defined structured query representation. This representation clarifies the target data entity to be queried, the filtering conditions, the aggregation operations, and the output format.

[0041] Subsequently, this structured query statement is input into a pre-built semantic metadata mapping model. Based on the mapping rules between global business concepts and local physical models stored therein, the model automatically associates the logical data entities and attributes mentioned in the query with the specific data tables, streams, or files actually stored on different edge nodes, thereby generating a clear cross-source data logical mapping relationship.

[0042] Based on this logical mapping relationship, the system automatically identifies all the data sources required to complete this query, analyzes the association paths and access order between these data sources, and constructs a visual data source dependency graph.

[0043] Next, the dependency graph of this data source is analyzed, and the entire query plan is divided according to the data source ownership and calculation type. Multiple query segments that can be executed independently or in groups are identified, forming a set of query segments.

[0044] To ensure the executability of each segment, the system performs semantic integrity checks on the set of query segments, verifying whether the data fields required by each segment exist and are accessible in its associated data source, eliminating segments with missing dependencies, and finally generating a series of validated, atomic query units.

[0045] Next, the system analyzes the logical relationships between these query units, determines which units' execution depends on the output of other units, and generates an execution dependency matrix describing the sequential dependencies between units.

[0046] Finally, the system performs static analysis or estimates based on historical information for each query unit to calculate its expected computational complexity, such as the amount of data involved and the complexity of the type of computational operation (e.g., join, aggregation). This computational complexity is combined with the execution dependency matrix for comprehensive analysis and grouping. Query units with dependencies that are suitable for merging are grouped into a subtask, and an initial scheduling priority sequence is determined for all generated subtasks. This sequence typically prioritizes subtasks on the critical path, those with high computational load, or those that produce intermediate results required by other tasks.

[0047] In a healthcare setting, clinical researchers input a raw query request through an analytics interface. For example, they might request statistics on the average age and admission diagnosis distribution of patients in all ICU wards who experienced hypoxemia and were simultaneously on ventilator support within the past 24 hours. Upon receiving this request, the system parses it and transforms it into a structured query statement, specifying multiple data entities to be associated, including basic patient information, vital sign timing events, ventilator status, and diagnostic codes.

[0048] The structured query is then input into a pre-built semantic metadata mapping model for the healthcare field. Based on its stored mapping rules, the model associates logical concepts such as ICU wards, hypoxemia events, and ventilator support with data sources actually stored on different edge nodes, such as inpatient information databases, real-time data streams from monitors, ventilator equipment logs, and diagnostic coding libraries, thereby generating a cross-source data logical mapping relationship.

[0049] Based on this logical mapping relationship, the system then identifies that completing this query requires data to be obtained from three edge nodes: the hospital data center node that stores the patient's basic information, the local edge server node responsible for the intensive care unit, and the ventilator equipment management node. Based on this, a data source dependency graph describing the data flow and dependencies is generated.

[0050] Based on this diagram, the system identifies the following query segments: the first segment filters and extracts relevant patient demographic information from the hospital data center node; the second segment queries events with blood oxygen saturation below a threshold within a specific time window from the ICU edge server node; and the third segment queries the ventilator start / stop status records within the same time period from the equipment management node. These three segments constitute the query segment set.

[0051] The system performs semantic integrity checks on the above set of fragments, confirming that the data fields required for each fragment (such as patient ID, timestamp, and device number) exist and can be associated in its associated data source, thereby generating three query units that pass the checks.

[0052] Subsequently, the system determines the execution order constraints between query units: the patient list (unit one) must be obtained first before query events (unit two and unit three) can be associated based on it. Unit two and unit three are independent and can be executed in parallel. Based on this, an execution dependency matrix is ​​generated.

[0053] The system calculates the computational complexity of each query unit. For example, Unit 1 involves an index scan of a large database, while Units 2 and 3 involve window aggregation calculations on time-series streaming data. Based on the execution dependency matrix, the system packages Units 2 and 3 into a parallel-executable subtask A, defines Unit 1 as subtask B, and determines a scheduling priority sequence: subtask B is executed first, and its output is used as input to subtask A to minimize the overall execution time.

[0054] In the fintech sector, an analyst submits a raw query request: to find all transfer transactions initiated by newly registered users within the past hour, where the transaction amount exceeds a set threshold and the recipient is located in a high-risk region. Upon receiving the request, the system generates a structured query statement, specifying the dimensions to be considered, including user attributes, transaction history, and a regional risk database.

[0055] The structured query is then input into a semantic metadata mapping model for the financial domain. The model maps logical attributes such as user registration time, transaction amount, and region code to the actual user center database, real-time transaction flow processing nodes, and external risk information database, generating cross-source data logical mapping relationships.

[0056] Based on this relationship, the system identifies the data sources involved in the query, which are located in multiple independent nodes such as the user center, the real-time transaction processing platform, and external data service interfaces, and generates a data source dependency graph describing their topological connections.

[0057] Based on this graph, the system then identified key query segments: filtering newly registered users from the user center; filtering large transactions from the real-time transaction stream; obtaining a list of high-risk areas from external interfaces; and finally performing correlation matching. These segments constitute a set of query segments.

[0058] Next, a semantic integrity check is performed to confirm that all required association keys (such as user ID and region code) for the fragments are available, generating four query units that pass the check.

[0059] Obtaining the list of high-risk areas (Unit 3) can be performed in parallel with filtering new users (Unit 1); however, filtering large transactions (Unit 2) and the final association matching (Unit 4) can only be performed after the results of the preceding units are produced. Based on this, an execution dependency matrix is ​​generated.

[0060] The complexity of each unit was then assessed: Units 1 and 3 involve lightweight queries; Unit 2 involves complex event processing of high-speed data streams, resulting in a heavy computational load; Unit 4 involves multi-table joins, leading to significant memory consumption. Based on the dependency matrix, the system divides Units 1 and 3 into parallelizable subtask 1, Unit 2 into subtask 2, and Unit 4 into subtask 3. The generated scheduling priority sequence is as follows: Subtask 1 is triggered simultaneously, Subtask 2 is triggered immediately after its partial results are ready, and finally Subtask 3 is executed, to achieve low-latency detection of risk events.

[0061] In some embodiments, calculating the computational complexity of each query unit and combining the computational complexity with the execution dependency matrix to generate multiple subtasks and corresponding scheduling priority sequences includes: Key features of each query unit are extracted, including data scan volume, computational operation density, memory access pattern, and I / O demand intensity. Data scan volume is estimated by analyzing query predicate conditions and data distribution statistics, predicting the product of the number of rows and columns to be accessed, and then corrected to the actual physical scan range based on data partitioning and indexing. Computational operation density is quantified based on the type, number, and nesting depth of operators in the query execution plan, assigning basic weight coefficients to different operators. For example, hash join operations have a weight of 3.8, sorting and aggregation operations have a weight of 3.2, and window functions have a weight of 4.5. The exponential growth in complexity caused by operator nesting is also considered, with each additional layer of nesting increasing complexity by a factor of 1.3. Memory access pattern is predicted by simulating the size of the intermediate result set, sorting buffer requirements, and hash table construction process in the execution plan, forecasting peak memory requirements and cache hit rate. When the estimated memory requirement exceeds 80% of the node capacity, the complexity score is increased by an additional 25%. I / O demand intensity is calculated based on the serialization size of intermediate results and disk access patterns. Random I / O operations are given a weight of 1.8 times, while sequential I / O operations maintain the baseline weight. For every 1GB increase in network data transfer, the complexity increases by 8 points. The four dimensions are normalized to the 0-1 range using Min-Max, and the entropy weight method is applied to dynamically determine the weight of each dimension, generating a comprehensive complexity score of 0-100. For query units involving deep learning inference, additional adjustment factors for the number of model parameters and the depth of the computational graph are added, with an additional 3 points added to the base score for every million parameters, forming an N×1 dimension computational complexity vector, where N is the total number of query units.

[0062] The execution dependency matrix is ​​an N×N sparse matrix constructed by analyzing the data flow dependencies and control flow constraints between query units. Matrix element M[i][j] represents the dependency strength of query unit i on j, with values ​​ranging from 0 to 1, where 0 indicates no dependency and 1 indicates strong dependency. Dependency strength is calculated based on three factors: data transmission volume ratio (weight 0.4), time-series constraint strictness (weight 0.35), and business criticality (weight 0.25). The data transmission volume ratio is calculated as the ratio of dependent data volume to the total output of the source unit; the time-series constraint strictness is determined based on time window requirements and causal relationship strength; business criticality is derived from query semantic analysis, such as financial risk control or medical diagnosis queries, which are automatically assigned higher criticality weights. The computational complexity vector is fused with the execution dependency matrix, using a multi-objective optimization function: PriorityScore[i] = α×ComplexityScore[i] + β×DependencyCriticism[i] + γ×DataLocalityFactor[i].

[0063] Here, PriorityScore[i] represents the overall priority score of the i-th query unit, which determines the task scheduling order; the higher the value, the higher the priority. α is the computational complexity weight coefficient, which controls the scheduling tendency of computationally intensive tasks under the overall system load pressure. The default value is 0.45, and its dynamic adjustment mechanism is based on the average utilization of the cluster CPU. When the utilization exceeds the 80% threshold, α increases by 0.1 to prioritize scheduling high-complexity tasks to idle resources. ComplexityScore[i] is the normalized computational complexity score of the i-th query unit, ranging from 0 to 100, comprehensively reflecting four dimensions: data scan volume, computational operation density, memory access pressure, and I / O demand intensity. β is the dependency criticality weight coefficient, reflecting the system's sensitivity to task dependencies. The default value is 0.4. When the proportion of latency-sensitive queries exceeds 30%, β increases by 0.15 to strengthen the priority execution of critical path tasks. DependencyCriticism[i] represents the criticality of the i-th query unit in the execution dependency graph. It is obtained by calculating the weighted sum of the i-th row and column in the execution dependency matrix, with a value ranging from 0 to 1. The closer the value is to 1, the more critical the unit is on the execution path, and its latency will directly affect the overall query completion time. γ is the data locality weight coefficient, with a default value of 0.15, used to balance the impact of data movement costs on scheduling decisions. When network bandwidth utilization exceeds a threshold, γ is dynamically increased to reduce cross-node data transmission. DataLocalityFactor[i] evaluates the physical proximity of the data required by the i-th query unit to the candidate execution node, using a normalized value ranging from 0 to 1. 1 indicates that all data is available locally on the target node, and 0 indicates that data needs to be accessed remotely. The three weight coefficients satisfy the constraint α+β+γ=1 to ensure the normalization characteristics of the priority score. At the same time, the dynamic weight adjustment mechanism adapts to changes in the system's runtime state, enabling the scheduling decision to achieve the best balance among the three objectives of computational load balancing, critical path optimization, and data movement minimization.

[0064] Based on the PriorityScore vector, the system applies an improved topology sorting algorithm to generate a scheduling sequence, while also considering subtask partitioning optimization. The algorithm maintains a ready queue, adding a query unit only when all its preceding dependent units are completed, and sorting them in descending order of PriorityScore; for units with the same score, sorting them in ascending order of data locality factor. The subtask generation process implements adaptive granularity control. For query units with a complexity score exceeding 90, a vertical partitioning strategy is applied, splitting them into multiple subtasks according to data partition boundaries or computational logic boundaries, with each subtask's complexity controlled within the range of 60-80 points. For units with scores between 50-90, the original granularity is maintained; for units below 50, a horizontal merging strategy is applied, merging adjacent units with the same data source dependency and consecutive execution order into a single subtask, controlling the post-merging complexity to not exceed 85 points. Each split or merged subtask inherits the dependencies of the original query unit and establishes new dependency constraints between subtasks to ensure the correctness of the execution logic. The scheduling priority sequence generation takes into account node resource constraints and timing windows. The system maintains a global resource schedule, recording the expected resource availability of each edge node at different time periods. Simulated scheduling verifies the resource sufficiency of each subtask within its estimated completion time window. For resource-conflicting subtasks, priorities are adjusted or target nodes are reassigned. The final output is a complete scheduling scheme containing subtask IDs, target node assignments, a list of dependent subtasks, estimated execution times, and priority numbers (1 being the highest). This scheme supports dynamic adjustment at runtime. When abnormal node performance or changes in data distribution are detected, a local rescheduling mechanism is triggered, recalculating only the priorities and allocation schemes of the affected subtasks to maintain overall scheduling stability and execution continuity.

[0065] In some embodiments, the method for constructing a semantic metadata mapping model includes: Collect structured and unstructured metadata from local data sources in edge nodes to generate a raw metadata set; The original metadata set is standardized in terms of data type and naming conventions to generate a unified format metadata dataset; Analyze the semantic relationships in the unified format metadata set, establish triples, and generate a basic semantic network; Based on the basic semantic network, synonyms, near-synonyms, and equivalence relations of local data sources are identified, and a semantic equivalence mapping table is generated. By integrating semantic equivalence mapping tables with basic semantic networks, an extensible semantic ontology is constructed, generating a domain knowledge graph. Hierarchical clustering and semantic abstraction are performed on the domain knowledge graph to extract common semantic patterns and generate a semantic pattern template library; The semantic pattern template library is associated with the physical location information of the local data source to establish a logical-physical mapping relationship and generate a data location mapping index. By combining the data location mapping index with the semantic pattern template library, a query transformation rule set is defined to generate a semantic transformation rule library; By integrating domain knowledge graphs, semantic pattern template libraries, data location mapping indexes, and semantic transformation rule bases, a semantic metadata mapping model is constructed.

[0066] This embodiment ensures data format uniformity and consistency by collecting and standardizing multi-source metadata, laying the foundation for subsequent semantic analysis. Next, by establishing a semantic network and identifying equivalence relations, it effectively addresses the semantic gap between heterogeneous data sources, improving data interoperability. The process of constructing a knowledge graph and extracting a semantic pattern template library enables the model to capture deep relationships between data, supporting complex cross-source queries. Associating physical location information and defining query rules further optimizes the data access path and reduces query latency. The entire construction process is tightly integrated, with preceding outputs serving as subsequent inputs. The resulting model possesses high scalability and adaptability, dynamically responding to diverse data environments and significantly improving the accuracy and efficiency of query parsing.

[0067] In the healthcare field, metadata from electronic medical records, medical images, laboratory tests, and vital sign monitoring devices is collected from edge nodes of various hospitals. This includes structured field descriptions and unstructured clinical document descriptions, generating a raw medical metadata set. This set covers diverse metadata such as patient basic information tables, diagnostic records, examination report templates, and device data format specifications.

[0068] Next, the raw metadata undergoes medical standardization processing, mapping diagnostic names to the ICD-11 coding system, unifying laboratory indicators to the LOINC standard, converting measurement units to international metrological standards, and generating a unified format metadata dataset that conforms to medical industry standards. This process resolves the naming inconsistencies between different hospitals, such as standardizing "myocardial infarction," "myocardial infarction," and "MI" into standardized terms.

[0069] Then, based on a unified format metadata dataset, the system analyzes the relationships between medical concepts, establishes triplet relationships such as disease-symptom-treatment, patient-examination-physician, and drug-dosage-contraindications, and generates a basic medical semantic network. This network clearly identifies the clinical association between hypertension and electrocardiogram abnormalities, the causal relationship between specific drugs and changes in laboratory indicators, and forms a preliminary medical knowledge structure.

[0070] Further analysis of the basic semantic network can identify synonyms and equivalence relationships in different medical systems. For example, it can establish a mapping between different patient ID systems used by various hospitals, and map the imaging examination codes such as CT and MRI used in radiology to clinical terms, generating a medical semantic equivalence mapping table to solve cross-institutional data understanding barriers.

[0071] Subsequently, the semantic equivalence mapping table is integrated with the basic semantic network, combining clinical guideline knowledge and medical literature relationships to construct an extensible medical semantic ontology, generating a medical domain knowledge graph covering multiple dimensions such as disease, symptoms, treatment, and prognosis. This graph supports concept expansion, allowing new diseases to be quickly incorporated into the system.

[0072] Next, hierarchical clustering is performed on the medical knowledge graph to categorize similar disease patterns and treatment pathways, extracting common semantic patterns such as chronic disease management, preoperative assessment, and postoperative follow-up, and generating a medical semantic pattern template library. These templates capture data association patterns in specific clinical scenarios, accelerating subsequent query parsing.

[0073] Then, the semantic pattern template library is associated with the physical location of each hospital's information system, recording the storage location of electronic medical records, the access path of the image archiving system, and the interface address of laboratory results. A logical-physical mapping relationship between clinical concepts and physical storage is established, a medical data location mapping index is generated, and the query execution path is optimized.

[0074] Next, by combining the data location mapping index and the semantic pattern template library, rules are defined such as converting a patient full view query into a multi-system joint query and converting a efficacy analysis query into a time-series data aggregation, generating a medical semantic transformation rule library, so that complex clinical queries can be automatically decomposed into executable operations.

[0075] Ultimately, the system integrates medical knowledge graphs, semantic pattern template libraries, data location mapping indexes, and semantic transformation rule bases into a unified medical semantic metadata mapping model. This model supports cross-hospital and cross-system clinical data queries, enabling seamless integration of patient information and providing comprehensive data support for clinical decision-making.

[0076] In the fintech field, the system collects metadata from various branches and edge nodes of business systems, including core transaction systems, customer relationship management, risk assessment models, market data interfaces, and compliance audit logs. This metadata includes structured database schemas and unstructured business rule documents, generating a raw financial metadata set. This set covers metadata descriptions such as account information, transaction history, customer profiles, market indicators, and regulatory rules.

[0077] Then, the raw metadata undergoes financial standardization processing, mapping product types to standard financial product classifications, unifying risk indicators to Basel Accord standards, and converting currency units to base currencies, generating a unified format metadata dataset that conforms to financial industry standards. This process resolves terminology differences between different business systems, such as unifying the classification of personal loans, consumer credit, and retail loans.

[0078] Based on a unified format metadata dataset, the system analyzes the relationships between financial concepts, establishing triplet relationships such as customer-account-transaction, product-risk-return, and market-volatility-exposure, generating a basic financial semantic network. This network clearly identifies the connection between large transactions and anti-money laundering rules, and the quantitative relationship between market volatility and portfolio risk, forming a preliminary financial knowledge structure.

[0079] Next, we delve into the basic semantic network to identify synonyms and equivalence relationships in different financial systems. For example, we establish a mapping between different customer numbering systems used by branches in different regions, correspond front-end business terms with back-end accounting subjects, and generate a financial semantic equivalence mapping table to solve cross-departmental data understanding barriers.

[0080] Subsequently, the semantic equivalence mapping table is integrated with the basic semantic network, combining regulatory policy knowledge and financial product characteristics to construct an extensible financial semantic ontology, generating a financial knowledge graph covering multiple dimensions such as customers, products, risks, and compliance. This graph supports the rapid inclusion of new financial products into the risk assessment system.

[0081] Next, the financial knowledge graph is hierarchically clustered to categorize similar transaction patterns and risk characteristics, extracting common semantic patterns such as high-frequency transaction monitoring, large-amount transfer warnings, and market risk transmission, generating a financial semantic pattern template library. These templates capture data correlation patterns in specific financial scenarios, accelerating risk query processing.

[0082] Then, the semantic pattern template library is associated with the physical location of each business system, recording the location of the transaction database server, the customer master data access interface, and the real-time risk control engine call path. This establishes a logical-physical mapping relationship between financial concepts and physical storage, generates a financial data location mapping index, and optimizes the risk control query response time.

[0083] Next, by combining the data location mapping index and the semantic pattern template library, rules are defined such as converting a full view query of a customer into a joint query of multiple systems and converting a risk aggregation query into a distributed computing task, thereby generating a financial semantic transformation rule library, which enables complex risk control queries to be automatically decomposed into efficient execution plans.

[0084] Ultimately, the system integrates financial knowledge graphs, semantic pattern template libraries, data location mapping indexes, and semantic transformation rule bases into a unified financial semantic metadata mapping model. This model supports cross-business and cross-regional data queries, enables real-time risk monitoring and assessment, provides immediate data insights for financial decision-making, and meets stringent data compliance requirements.

[0085] S103: Obtain the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; In this embodiment, the status information of the edge nodes is obtained, and a delay-aware scheduling algorithm is used to allocate subtasks to each edge node, generating a task allocation scheme including: Real-time acquisition of state information from each edge node generates a real-time state vector; Receive subtasks and obtain the computational complexity and data dependencies from the subtasks; The real-time state vector is matched with the computational complexity using multi-dimensional features to generate a node task fit matrix. Based on the node task adaptation matrix and preset delay weight parameters, a delay-aware scheduling algorithm is applied to generate an initial task allocation scheme. Analyze the data dependencies and execution order constraints between subtasks to generate a task execution dependency graph; The task execution dependency graph and the initial task allocation scheme are used to detect and resolve conflicts, and an intermediate task allocation scheme is generated. Obtain historical performance fluctuation data of edge nodes, and make robust adjustments to the intermediate task allocation scheme based on the historical performance fluctuation data to generate a task allocation scheme.

[0086] This embodiment ensures that scheduling decisions are based on the latest system conditions by collecting edge node status information in real time and generating state vectors, thus improving real-time response. By analyzing the computational complexity and data dependencies of subtasks and combining multi-dimensional feature matching to generate a fit matrix, task allocation is made more closely aligned with the actual capabilities of nodes, optimizing resource utilization. The combination of a latency-aware scheduling algorithm and preset weight parameters effectively balances latency constraints and system load, generating an initial scheme. Further consideration is given to data dependencies and execution order between tasks, and conflict detection and resolution are performed to ensure the correctness and efficiency of task execution. Finally, historical performance fluctuation data is introduced for robustness adjustments, enhancing the scheme's adaptability to dynamic environments and reducing the impact of performance fluctuations. The entire process is interconnected, achieving efficient and reliable task allocation.

[0087] In the healthcare field, edge nodes such as emergency departments, testing centers, pharmacies, and intensive care units are selected as task execution carriers. Latency data, current load rate, bandwidth usage, and remaining computing resources of each node are collected in real time and quantified according to a unified dimension to generate a real-time status vector containing four indicators.

[0088] Next, it receives sub-tasks for patient data association analysis, including patient screening, blood glucose data extraction, medication record query, and test result matching. It extracts the computational complexity from each sub-task (blood glucose data extraction has the highest complexity due to its involvement of continuous time series data; patient screening has the smallest amount of data processing and the lowest complexity). At the same time, it clarifies the data dependency relationship that the patient screening sub-task must be executed first, and the other three sub-tasks depend on its output results.

[0089] Then, the real-time state vectors of each edge node are matched with the computational complexity of the subtasks using multi-dimensional features. The emergency department node has the lowest latency and moderate load, and is the most suitable for the high-complexity blood glucose data extraction subtask. The laboratory center node has the highest suitability for the test result matching subtask. Based on this, a node task suitability matrix is ​​generated.

[0090] Based on this adaptation matrix and combined with the high latency weight parameters preset in the medical scenario (prioritizing the real-time performance of data processing), a latency-aware scheduling algorithm is applied to allocate the blood glucose data extraction subtask to the emergency department node, the test result matching subtask to the testing center node, the medication record query subtask to the pharmacy node, and the patient screening subtask to the intensive care unit node with the lowest load, thus generating an initial task allocation plan.

[0091] Subsequently, the data dependencies and execution order constraints between subtasks were analyzed, and the patient screening subtask was identified as the starting node, while blood glucose data extraction, medication record query, and test result matching were identified as subsequent parallel nodes, thus generating a task execution dependency graph.

[0092] Next, conflict detection was performed between the task execution dependency graph and the initial plan. It was found that the intensive care unit node currently needs to prioritize the processing of monitoring data. If the patient screening sub-task is assigned, it may lead to increased latency. By adjusting, the patient screening sub-task is transferred to the edge unit attached to the testing center node with a lower load. After resolving the conflict, an intermediate task allocation plan is generated.

[0093] By continuously acquiring historical performance fluctuation data of each edge node over the past three months, it was found that the load of the testing center node fluctuates significantly during the peak testing period from 10:00 to 11:00 AM. Since the current period is not during the peak period, no adjustment is needed. The emergency department node has historically shown smaller fluctuations and strong stability. Based on this, the intermediate solution was robustly adjusted, and the final task allocation solution was generated.

[0094] In the field of fintech, bank offline branch terminals, core business edge nodes, security monitoring nodes, and risk control dedicated nodes are selected as execution nodes. The transmission latency, CPU load rate, bandwidth utilization and memory usage of each node are collected in real time and then quantified to generate a real-time status vector containing four indicators.

[0095] Then, it receives sub-tasks related to account risk screening, including account screening, transaction data extraction, login log query, and risk control record matching. It obtains the computational complexity from the sub-tasks (transaction data extraction involves massive transaction records, so it has the highest complexity; account screening has the lowest complexity). At the same time, it determines the data dependency relationship: the account screening sub-task is executed first, the transaction data extraction and login log query sub-tasks are executed in parallel, and the risk control record matching sub-task depends on the output results of the first three sub-tasks.

[0096] Then, the real-time state vectors of each node are matched with the computational complexity of the subtasks using multi-dimensional features. The core business edge nodes have sufficient computing resources and low bandwidth utilization, and are the most compatible with the transaction data extraction subtask. The security monitoring nodes are the most compatible with the login log query subtask, and the risk control dedicated nodes are the most compatible with the risk control record matching subtask, thus generating a node task compatibility matrix.

[0097] Next, based on this matrix and combined with the preset delay weight parameters of the financial scenario (prioritizing the timeliness of transaction data processing and the speed of risk control response), a delay-aware scheduling algorithm is applied to allocate the transaction data extraction sub-task to the core business edge node, the login log query sub-task to the security monitoring node, the risk control record matching sub-task to the risk control dedicated node, and the account screening sub-task to the offline branch terminal, thus generating an initial task allocation scheme.

[0098] Next, we analyze the dependencies and execution order constraints between subtasks and construct a task execution dependency graph that starts with account screening, runs in parallel with transaction data extraction and login log querying, and ends with risk control record matching.

[0099] Subsequently, conflict detection was performed between the dependency graph and the initial solution. It was found that the current network latency of the offline outlet terminal was slightly high. If the account screening subtask was assigned, it might affect the start of subsequent tasks. The account screening subtask was then moved to the idle processing unit of the core business edge node. After resolving the conflict, an intermediate task allocation solution was generated.

[0100] Historical performance fluctuation data of each node over the past six months was obtained. The core business edge nodes experienced significant load fluctuations during the peak trading hours of 3-4 pm on trading days, while the current period is off-peak. The security monitoring nodes showed stable historical fluctuations. Based on this, the intermediate solution was robustly adjusted, reserving 20% ​​of computing resources for the core business edge nodes to cope with sudden loads, and finally generating a task allocation plan.

[0101] In some embodiments, during the task distribution phase, the distributed task orchestrator first receives overall task information transmitted from the upper layer. This information includes the task objective, the range of data to be processed, subtask partitioning rules, and execution constraints. The orchestrator parses the overall task, breaking it down into multiple independent subtasks according to data processing logic and resource adaptation requirements, clarifying the computational requirements, data dependencies, and execution priorities of each subtask. Subsequently, the orchestrator obtains the real-time availability status of each edge node (including remaining computing resources, network connectivity, and current load), and generates an optimal task distribution list based on the subtask attributes. The list clearly specifies the target edge node, data transmission path, and execution start conditions for each subtask. Through a preset communication protocol, the orchestrator synchronously distributes the subtask execution instructions, data access permissions, and related configuration parameters to the corresponding edge nodes, ensuring that each node accurately receives the task requirements.

[0102] In the status monitoring phase, the distributed task orchestrator establishes a real-time monitoring channel, receiving subtask execution status data reported by each edge node at fixed time intervals. This data includes task execution progress, current computing resource utilization, data processing volume, and whether any anomalies have occurred. The orchestrator performs real-time parsing and verification of the reported status data. If an anomaly is detected, such as execution latency exceeding a threshold, insufficient resources, or data transmission interruption, an early warning mechanism is immediately triggered. Adjustment instructions are generated according to preset rules, such as transferring unfinished subtasks to other available nodes or temporarily allocating additional resources to that node. Simultaneously, the orchestrator summarizes the execution status of each subtask by task dimension, generating a global task progress view that clearly presents the overall task completion rate, the time consumed at each stage, and the node execution efficiency. After all tasks are completed, the orchestrator collects the execution result feedback from each node, verifies the task completion status, and generates a complete task execution status report, providing data support for subsequent task optimization.

[0103] S104: Based on the task allocation scheme and the local data source, execute the corresponding sub-tasks at each edge node to generate local results; In this embodiment, based on the task allocation scheme and local data source, corresponding subtasks are executed at each edge node to generate local results, including: Receive the task allocation scheme, which includes subtask calculation instructions and data access paths; Based on the data access path specified in the task allocation scheme, the real-time data stream from the local data source is obtained through the edge data proxy layer to generate the dataset to be processed. Load a lightweight analysis model that matches the subtask computation instructions to generate an edge computing execution environment; In the edge computing execution environment, perform data aggregation or model inference on the dataset to be processed to generate raw computation results; Feature extraction and data summarization are performed on the original calculation results to generate a simplified result set; Data anonymization is performed on the simplified result set to generate intermediate results; Add timestamps and data lineage identifiers to intermediate results to generate partial calculation results; The local calculation results are encapsulated into standardized data packets to generate local results.

[0104] This embodiment ensures that edge nodes can quickly locate and obtain the required local data through a clear task allocation scheme and data access path, significantly reducing data retrieval and preparation time and improving processing efficiency. Utilizing a lightweight analysis model to perform data aggregation or model inference locally effectively reduces the need for data transmission to the cloud, minimizing network bandwidth consumption and overall latency. By extracting features and summarizing data from the original calculation results, a concise result set is generated, greatly compressing the data volume without losing key information and further optimizing transmission efficiency. Introducing a data anonymization step removes or encrypts sensitive information before the data leaves the edge node, significantly enhancing data security and privacy protection capabilities. Adding timestamps and data lineage identifiers ensures the traceability of results, facilitating subsequent data quality control and problem investigation. Finally, the operation of encapsulating standardized data packets ensures that the results generated by different edge nodes have a consistent format and structure, simplifying the result integration process in the cloud. The entire process forms a complete data processing chain at the edge, achieving reasonable distribution of computing load and efficient utilization of resources.

[0105] In a healthcare scenario, an image diagnostic edge node in a regional medical center executes sub-tasks and generates local results according to the following steps. The node first receives a task allocation scheme from the cloud-edge collaborative control layer, which specifies that the sub-task is to automatically detect lung nodules in a batch of chest CT images and indicates that the data access path points to the hospital's image archiving system.

[0106] According to the path specified in the scheme, the edge data agent deployed on this node accesses the local image library, obtains the CT image data stream to be analyzed, and generates a dataset to be processed containing DICOM files of a specific sequence.

[0107] Subsequently, the node loads a pre-built lightweight deep learning model that matches the lung nodule detection task, generating a ready edge computing execution environment.

[0108] In this environment, model inference calculations are performed on the imported CT image dataset. The model analyzes the images frame by frame, identifies and labels potential nodule regions, and generates raw calculation results containing nodule coordinates, size, and probability.

[0109] Next, the system extracts and summarizes the features of these raw results, such as calculating the average size, maximum size and number of suspected malignant nodules in the batch of images, and generating a concise and simplified result set.

[0110] To protect patient privacy, the system desensitizes the simplified result set, removing all identifiers directly associated with specific patients, such as names and ID numbers, and generating intermediate results that retain only statistical features and anonymized image numbers.

[0111] Then, the system adds a precise timestamp to the intermediate result and injects a data lineage identifier to indicate which imaging device, which examination, and the model version it originated from, generating a traceable local calculation result.

[0112] Finally, this partial computation result, which includes statistical summaries, anonymization markers, and metadata, is packaged into a structured data package that conforms to medical data exchange standards, generating the final partial result ready to be uploaded to the cloud for aggregation analysis.

[0113] In a fintech scenario, an edge node of a branch's server executes a risk control sub-task according to the following steps: The node receives a task allocation plan, instructing it to perform real-time fraud scoring on the branch's card-swiping transactions over the past hour. The plan includes calculation instructions for the scoring model and an access path to the local transaction database.

[0114] According to this path, the node's edge data proxy layer connects to the branch transaction database, obtains encrypted transaction log records that meet the time range in real time, and generates a dataset to be processed.

[0115] The node loads a lightweight machine learning model corresponding to the fraud scoring instruction. This model has been optimized for local transaction patterns, thereby generating a ready-to-use edge computing execution environment.

[0116] In this environment, the system performs model inference on each transaction. The model calculates a fraud probability score in real time based on features such as transaction amount, location, and merchant type, generating raw calculation results containing the transaction ID and corresponding risk score.

[0117] The system then extracts and summarizes the features of these raw scores, such as calculating the proportion of high-risk transactions, listing the top five transactions with the highest risk scores, and generating a concise result set that highlights the key points.

[0118] Next, data anonymization is performed, masking the card number field in the transaction records and generalizing the names of sensitive merchants to generate intermediate results that meet both risk control analysis needs and privacy regulations.

[0119] The system adds a processing completion timestamp to the above intermediate results and embeds a data lineage identifier, recording the corresponding data time window, the risk control model version used, and the line code, generating local calculation results with complete context.

[0120] Finally, this local calculation result is encapsulated into a standardized data packet with a digital signature supported by the intranet secure transmission protocol, generating a local result that can be sent to the head office risk control center for global aggregation.

[0121] S105: Integrate the local results in the cloud to generate global analysis results.

[0122] Specifically, the cloud service receives standardized data packets from multiple edge nodes, each packet encapsulating a local computation result. The cloud first decapsulates and parses these data packets, extracting the core data content and metadata to generate a set of local results to be integrated.

[0123] The system then performs an integrity check on the received partial result set, verifying whether all expected subtask results have arrived and are in complete format based on the data lineage identifier and task identifier carried by each result. Results that pass the check are marked as valid and enter the subsequent processing queue.

[0124] Next, based on the metadata attached to each verified local result, especially its timestamp and data lineage, the system logically re-associates and aligns these results according to the structure of the original query task, restoring the complete data view of distributed execution and generating a structured intermediate data view.

[0125] Based on the aggregation or fusion logic defined by the final analysis objective, the system performs global calculations on the structured intermediate data view. This may include data aggregation and statistics across all edge nodes, correlation analysis, trend calculations, or model re-inference based on global information to generate preliminary global analysis results.

[0126] Subsequently, the system performs consistency checks and conflict resolution on the preliminary global analysis results. For example, it addresses the issue of slight misalignment of data time windows between edge nodes due to network latency, ensuring the accuracy and reliability of the results and generating a global result that has undergone consistency processing.

[0127] Finally, the system converts the processed global results into a specified output format, such as an analysis report, a visualized data structure, or a set of instructions that triggers subsequent business processes, thereby generating global analysis results that can be used by end users or upper-layer applications.

[0128] This embodiment also includes: Real-time data collection of system latency, network bandwidth utilization, and computing resource energy consumption at edge nodes generates a multi-dimensional performance index sequence. The time-series features of the multi-dimensional performance index sequence are extracted and normalized to generate system state feature vectors; The system state feature vector is input into a pre-trained reinforcement learning model to generate a set of candidate resource allocation actions; Evaluate the impact of candidate resource allocation action sets on system performance and generate an action value evaluation matrix; Based on the action value assessment matrix, the optimal resource allocation action is selected, and resource adjustment instructions are generated. Execute resource adjustment commands and monitor system response, generating policy execution feedback data; Acquire historical performance data, compare and analyze the strategy execution feedback data with the historical performance data, and generate a model training sample set; The policy network parameters of the reinforcement learning model are updated using the model training sample set to generate a resource allocation policy model. Resource allocation schemes are generated through a resource allocation strategy model.

[0129] This embodiment provides a comprehensive and accurate data foundation for resource allocation decisions by collecting multi-dimensional performance indicators in real time and generating system state feature vectors. Secondly, it utilizes a pre-trained reinforcement learning model to generate a candidate action set, achieving intelligent decision-making and adapting to complex dynamic environments. Next, by evaluating the system impact of candidate actions and selecting the optimal solution, the scientific and efficient nature of the resource allocation strategy is ensured. The mechanism of executing resource adjustment commands and monitoring system responses forms an effective feedback loop, enabling the system to self-correct. By comparing and analyzing real-time feedback data with historical performance data and generating a training sample set to update model parameters, continuous learning and optimization of the system are achieved.

[0130] In healthcare scenarios, adaptive resource allocation optimization is implemented as follows: Edge computing nodes located in different hospitals collect real-time data on system latency during medical image analysis processing, network bandwidth utilization with the regional medical cloud center, and energy consumption of GPU servers, forming a continuous series of multi-dimensional performance indicators.

[0131] The system extracts features from these time-series indicators, such as calculating the moving average of latency, the variance of bandwidth utilization, and energy consumption trends, and performs normalization processing to generate a system state feature vector that comprehensively reflects the current load and efficiency.

[0132] This feature vector is fed into a reinforcement learning model specifically designed for medical task scheduling. Based on the current state, the model outputs a set of candidate resource allocation actions, such as dynamically adjusting the ratio of CPU cores allocated to AI diagnosis tasks and medical record retrieval tasks, or switching to a low-power operating mode.

[0133] The system evaluates the impact of each candidate action on overall performance in a simulated environment, predicts the possible changes in end-to-end latency, bandwidth usage, and energy consumption, and generates a quantitative action value evaluation matrix.

[0134] Based on this matrix, the system selects the action with the highest evaluation value, namely, prioritizing AI diagnostic resources for critical cases while limiting the concurrency of non-urgent retrieval tasks, and generates specific resource adjustment instructions. The system executes the instruction and monitors the actual delay and energy consumption changes in the image analysis process in real time after the instruction is executed, collects system response data, and generates strategy execution feedback data.

[0135] The system retrieves performance data under similar load patterns from the historical database, compares it with the current feedback data, analyzes the actual effect of the strategy adjustment and the gap with expectations, and generates a training sample set for model improvement.

[0136] Using these new training samples, the policy network parameters of the reinforcement learning model are incrementally updated, its decision logic is fine-tuned, and an evolved next-generation resource allocation policy model is generated.

[0137] Ultimately, in actual operation, the system continuously generates dynamic resource allocation schemes through this optimized model, ensuring that medical edge computing resources always prioritize serving latency-sensitive emergency intelligent diagnostic tasks.

[0138] In fintech scenarios, adaptive resource allocation optimization is implemented as follows: Monitoring modules deployed at each transaction processing edge node collect real-time data on pipeline latency for high-frequency transaction risk control checks, bandwidth utilization when reporting results to the head office data center, and real-time energy consumption of computing devices, generating a high-frequency, multi-dimensional performance indicator sequence.

[0139] The system processes these sequences in real time, extracts key time-series features such as latency peaks, bandwidth saturation periods, and power consumption curves, and standardizes them to generate a system state feature vector describing the current transaction processing environment.

[0140] The state feature vector is fed into a reinforcement learning model trained for low-latency financial scenarios. Based on the current state, the model generates a series of candidate actions, such as adjusting memory allocation between stream processing and batch risk scanning tasks, or dynamically enabling backup network links.

[0141] The system evaluates the impact of each candidate action on key business metrics, including transaction processing throughput, risk event detection latency, and overall energy efficiency, forming an action value assessment matrix that includes multi-dimensional revenue predictions.

[0142] Based on this matrix, the system selects the optimal action, such as allocating more resources to stream processing during high-concurrency periods at market opening to ensure low latency, and generates explicit resource adjustment instructions.

[0143] The system executes the instruction and closely monitors the processing delay and risk control coverage of subsequent transactions, collecting actual operational data as strategy execution feedback data.

[0144] The system acquires performance data from the same period in history or under similar market fluctuations, compares and analyzes the current feedback with historical performance, quantifies the benefits of this strategy adjustment, and generates a new model training sample set.

[0145] Using this sample set, the decision network of the reinforcement learning model is updated online to optimize its decision accuracy under complex market conditions, generating a more adaptive resource allocation strategy model.

[0146] In daily operations, the system relies on this model to continuously generate and execute resource allocation plans, thereby intelligently balancing processing speed, risk control accuracy, and operating costs under different conditions such as peak trading periods and network fluctuations.

[0147] Please see Figure 3 This embodiment provides a low-latency data collaborative processing device 300, including: Configuration unit 301 is used to configure data proxies on multiple edge nodes and access local data sources; Parsing unit 302 is used to parse the query task into multiple subtasks through a semantic metadata mapping model in response to the query task; The allocation unit 303 is used to acquire the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load and bandwidth. Execution unit 304 is used to execute the corresponding sub-tasks at each edge node according to the task allocation scheme and the local data source, and generate local results; Integration unit 305 is used to integrate the local results in the cloud to generate global analysis results.

[0148] Furthermore, the configuration unit 301 includes: The agent configuration subunit is used to configure lightweight data agents for multiple edge nodes and determine the local data source type corresponding to each edge node. The data source access subunit is used to access local data sources of corresponding types based on the lightweight data proxy configured for each edge node. The local data sources include sensor streams, business databases, and log systems. The engine call subunit is used to call the lightweight data virtualization engine through the lightweight data proxy of each edge node, to perform unified access adaptation to the connected local data source, and generate data source access adaptation results. The data summarization subunit is used at each edge node to perform data summarization and feature extraction on the local data source based on the data source access adaptation result, and generate a data feature summary. The real-time monitoring subunit is used to monitor data change events of the connected local data source in real time, and output data change correlation information in combination with the data feature summary.

[0149] Furthermore, the parsing unit 302 includes: The request receiving subunit is used to receive the user's original query request and generate a structured query statement; The expression input subunit is used to input the structured query expression into the semantic metadata mapping model to generate cross-source data logical mapping relationships; The relationship identification subunit is used to identify the topological structure of the data sources involved in the query based on the cross-source data logical mapping relationship, and generate a data source dependency graph; The fragment identification subunit is used to identify query fragments based on the data source dependency graph and generate a query fragment set. The integrity verification subunit is used to perform semantic integrity verification on the set of query fragments and generate query units that pass the verification. A constraint determination subunit is used to determine the execution order constraints between the query units and generate an execution dependency matrix. The sequence generation subunit is used to calculate the computational complexity of each query unit, and combines the computational complexity with the execution dependency matrix to generate multiple subtasks and corresponding scheduling priority sequences.

[0150] Furthermore, the method for constructing the semantic metadata mapping model includes: Collect structured and unstructured metadata from local data sources in edge nodes to generate a raw metadata set; The original metadata set is subjected to data type standardization and naming convention processing to generate a unified format metadata dataset; Analyze the semantic relationships in the unified format metadata set, establish triples, and generate a basic semantic network; Based on the aforementioned basic semantic network, synonyms, near-synonyms, and equivalence relations of local data sources are identified, and a semantic equivalence mapping table is generated. The semantic equivalence mapping table is fused with the basic semantic network to construct an extensible semantic ontology and generate a domain knowledge graph. Hierarchical clustering and semantic abstraction are performed on the domain knowledge graph to extract common semantic patterns and generate a semantic pattern template library; The semantic pattern template library is associated with the physical location information of the local data source to establish a logical-physical mapping relationship and generate a data location mapping index. By combining the data location mapping index with the semantic pattern template library, a query transformation rule set is defined, and a semantic transformation rule library is generated; The domain knowledge graph, the semantic pattern template library, the data location mapping index, and the semantic transformation rule library are integrated to construct a semantic metadata mapping model.

[0151] Furthermore, the allocation unit 303 includes: The real-time acquisition subunit is used to collect the status information of each edge node in real time and generate a real-time status vector. The subtask receiving subunit is used to receive subtasks and obtain the computational complexity and data dependencies from the subtasks; A multi-dimensional feature matching subunit is used to perform multi-dimensional feature matching between the real-time state vector and the computational complexity to generate a node task adaptation matrix. The scheme generation subunit is used to generate an initial task allocation scheme based on the node task adaptability matrix and preset delay weight parameters, applying the delay-aware scheduling algorithm. The relationship analysis subunit is used to analyze the data dependencies and execution order constraints between the subtasks and generate a task execution dependency graph. The conflict detection subunit is used to detect and resolve conflicts between the task execution dependency graph and the initial task allocation scheme, and generate an intermediate task allocation scheme. The robust adjustment subunit is used to acquire historical performance fluctuation data of the edge nodes and perform robust adjustment of the intermediate task allocation scheme based on the historical performance fluctuation data to generate a task allocation scheme.

[0152] Furthermore, the execution unit 304 includes: The scheme receiving subunit is used to receive the task allocation scheme, which includes subtask calculation instructions and data access paths; The dataset generation subunit is used to obtain real-time data streams from local data sources through the edge data proxy layer according to the data access path specified in the task allocation scheme, and generate a dataset to be processed. The model loading subunit is used to load a lightweight analysis model that matches the computation instructions of the subtask and generate an edge computing execution environment. The model inference subunit is used to perform data aggregation or model inference on the dataset to be processed in the edge computing execution environment to generate raw calculation results. The feature extraction subunit is used to extract features and summarize data from the original calculation results to generate a simplified result set. The desensitization processing subunit is used to perform data desensitization processing on the simplified result set and generate intermediate results; The lineage identification subunit is used to add timestamps and data lineage identifiers to the intermediate results and generate local calculation results; The encapsulation subunit is used to encapsulate the local calculation results into a standardized data packet to generate local results.

[0153] Furthermore, it also includes: The data acquisition unit is used to collect data on system latency, network bandwidth utilization, and computing resource energy consumption of edge nodes in real time, and generate a multi-dimensional performance index sequence. The time-series feature extraction unit is used to extract and normalize the time-series features of the multi-dimensional performance index sequence to generate a system state feature vector. The vector input unit is used to input the system state feature vector into a pre-trained reinforcement learning model to generate a set of candidate resource allocation actions. The impact assessment unit is used to assess the impact of the candidate resource allocation action set on system performance and generate an action value assessment matrix. An action selection unit is used to select the optimal resource allocation action based on the action value evaluation matrix and generate a resource adjustment instruction. The response monitoring unit is used to execute the resource adjustment instructions and monitor the system response, and generate policy execution feedback data. The comparative analysis unit is used to acquire historical performance data, compare and analyze the strategy execution feedback data with the historical performance data, and generate a model training sample set. The parameter update unit is used to update the policy network parameters of the reinforcement learning model using the model training sample set to generate a resource allocation policy model. The allocation scheme generation unit is used to generate a resource allocation scheme through the resource allocation strategy model.

[0154] This invention provides a low-latency data collaborative processing device. First, data proxies are configured at multiple edge nodes to access local data sources. In response to a query task, the query task is parsed into multiple sub-tasks using a semantic metadata mapping model. The status information of the edge nodes is acquired, and a latency-aware scheduling algorithm is used to allocate the sub-tasks to each edge node, generating a task allocation scheme. The status information includes at least one of latency, load, and bandwidth. Based on the task allocation scheme and the local data source, the corresponding sub-tasks are executed at each edge node, generating local results. The local results are then integrated in the cloud to generate global analysis results. By configuring data proxies at edge nodes to access local data sources, in-situ data analysis is achieved. Simultaneously, the latency-aware scheduling algorithm reduces data transmission and centralized processing time, solving the high latency problem. Secondly, the task allocation scheme coordinates edge and central resources, avoiding resource idleness and overload. Thirdly, cross-source logical queries are implemented based on the semantic metadata mapping model, reducing the risk of privacy leakage and solving the problem of heterogeneous data integration. This significantly reduces data processing latency, improves resource utilization, and enhances data security and compliance.

[0155] Specific limitations regarding the low-latency data collaborative processing device can be found in the limitations of the low-latency data collaborative processing method described above, and will not be repeated here. Each unit in the aforementioned low-latency data collaborative processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0156] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a low-latency data collaborative processing method on the server side.

[0157] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a low-latency data collaborative processing method.

[0158] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Configure data proxies on multiple edge nodes to connect to local data sources; In response to a query task, the query task is parsed into multiple subtasks using a semantic metadata mapping model; Obtain the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; Based on the task allocation scheme and the local data source, the corresponding sub-tasks are executed at each edge node to generate local results; The local results are integrated in the cloud to generate global analysis results.

[0159] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Configure data proxies on multiple edge nodes to connect to local data sources; In response to a query task, the query task is parsed into multiple subtasks using a semantic metadata mapping model; Obtain the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; Based on the task allocation scheme and the local data source, the corresponding sub-tasks are executed at each edge node to generate local results; The local results are integrated in the cloud to generate global analysis results.

[0160] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0162] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0163] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A low-latency data collaborative processing method, characterized in that, include: Configure data proxies on multiple edge nodes to connect to local data sources; In response to a query task, the query task is parsed into multiple subtasks using a semantic metadata mapping model; Obtain the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth; Based on the task allocation scheme and the local data source, the corresponding sub-tasks are executed at each edge node to generate local results; The local results are integrated in the cloud to generate global analysis results.

2. The low-latency data collaborative processing method according to claim 1, characterized in that, The configuration of data proxies at multiple edge nodes and access to local data sources includes: Configure lightweight data proxies for multiple edge nodes and determine the local data source type for each edge node; Based on the lightweight data proxy configured on each edge node, it accesses the corresponding type of local data source, which includes sensor streams, business databases, and log systems; The lightweight data proxy at each edge node calls the lightweight data virtualization engine to perform unified access adaptation to the connected local data source and generate data source access adaptation results. At each edge node, based on the data source access adaptation results, data summarization and feature extraction are performed on the local data source to generate a data feature summary. The system monitors data change events from the connected local data source in real time and outputs data change correlation information based on the data feature summary.

3. The low-latency data collaborative processing method according to claim 1, characterized in that, The step of parsing the query task into multiple sub-tasks through a semantic metadata mapping model includes: Receive the user's original query request and generate a structured query statement; The structured query expression is input into the semantic metadata mapping model to generate a cross-source data logical mapping relationship; Based on the cross-source data logical mapping relationship, the topology of the data sources involved in the query is identified, and a data source dependency graph is generated; Based on the data source dependency graph, query fragments are identified and a set of query fragments is generated. Perform semantic integrity verification on the set of query fragments and generate query units that pass the verification; Determine the execution order constraints between the query units and generate an execution dependency matrix; The computational complexity of each query unit is calculated, and the computational complexity is combined with the execution dependency matrix to generate multiple subtasks and corresponding scheduling priority sequences.

4. The low-latency data collaborative processing method according to claim 1, characterized in that, The method for constructing the semantic metadata mapping model includes: Collect structured and unstructured metadata from local data sources in edge nodes to generate a raw metadata set; The original metadata set is subjected to data type standardization and naming convention processing to generate a unified format metadata dataset; Analyze the semantic relationships in the unified format metadata set, establish triples, and generate a basic semantic network; Based on the aforementioned basic semantic network, synonyms, near-synonyms, and equivalence relations of local data sources are identified, and a semantic equivalence mapping table is generated. The semantic equivalence mapping table is fused with the basic semantic network to construct an extensible semantic ontology and generate a domain knowledge graph. Hierarchical clustering and semantic abstraction are performed on the domain knowledge graph to extract common semantic patterns and generate a semantic pattern template library; The semantic pattern template library is associated with the physical location information of the local data source to establish a logical-physical mapping relationship and generate a data location mapping index. By combining the data location mapping index with the semantic pattern template library, a query transformation rule set is defined, and a semantic transformation rule library is generated; The domain knowledge graph, the semantic pattern template library, the data location mapping index, and the semantic transformation rule library are integrated to construct a semantic metadata mapping model.

5. The low-latency data collaborative processing method according to claim 1, characterized in that, The process of acquiring the state information of edge nodes and using a delay-aware scheduling algorithm to allocate the subtasks to each edge node, generating a task allocation scheme, includes: Real-time acquisition of state information from each edge node generates a real-time state vector; Receive subtasks and obtain the computational complexity and data dependencies from the subtasks; The real-time state vector is matched with the computational complexity using multi-dimensional feature matching to generate a node task fit matrix. Based on the node task adaptation matrix and the preset delay weight parameters, the delay-aware scheduling algorithm is applied to generate an initial task allocation scheme. Analyze the data dependencies and execution order constraints among the subtasks to generate a task execution dependency graph; The task execution dependency graph is subjected to conflict detection and resolution with the initial task allocation scheme to generate an intermediate task allocation scheme. The historical performance fluctuation data of the edge nodes is obtained, and the intermediate task allocation scheme is robustly adjusted based on the historical performance fluctuation data to generate a task allocation scheme.

6. The low-latency data collaborative processing method according to claim 1, characterized in that, The step of executing the corresponding sub-tasks at each edge node according to the task allocation scheme and the local data source to generate local results includes: Receive a task allocation scheme, which includes subtask calculation instructions and data access paths; According to the data access path specified in the task allocation scheme, the real-time data stream of the local data source is obtained through the edge data proxy layer to generate the dataset to be processed; Load a lightweight analysis model that matches the computation instructions of the subtask to generate an edge computing execution environment; In the edge computing execution environment, data aggregation or model inference is performed on the dataset to be processed to generate raw calculation results; Feature extraction and data summarization are performed on the original calculation results to generate a simplified result set; The simplified result set is then subjected to data anonymization processing to generate intermediate results; Add timestamps and data lineage identifiers to the intermediate results to generate local calculation results; The local calculation results are encapsulated into a standardized data packet to generate local results.

7. The low-latency data collaborative processing method according to claim 1, characterized in that, Also includes: Real-time data collection of system latency, network bandwidth utilization, and computing resource energy consumption at edge nodes generates a multi-dimensional performance index sequence. The multi-dimensional performance index sequence is subjected to time-series feature extraction and normalization to generate a system state feature vector; The system state feature vector is input into a pre-trained reinforcement learning model to generate a set of candidate resource allocation actions; Evaluate the impact of the candidate resource allocation action set on system performance and generate an action value evaluation matrix; Based on the action value evaluation matrix, the optimal resource allocation action is selected, and a resource adjustment instruction is generated. Execute the resource adjustment instructions and monitor the system response to generate policy execution feedback data; Acquire historical performance data, and compare and analyze the strategy execution feedback data with the historical performance data to generate a model training sample set; The model training sample set is used to update the policy network parameters of the reinforcement learning model to generate a resource allocation policy model; The resource allocation strategy model is used to generate a resource allocation scheme.

8. A low-latency data collaborative processing device, characterized in that, include: The configuration unit is used to configure data proxies across multiple edge nodes and connect to local data sources; The parsing unit is used to parse the query task into multiple subtasks in response to the query task through a semantic metadata mapping model; The allocation unit is used to acquire the status information of the edge nodes, use a latency-aware scheduling algorithm to allocate the subtasks to each edge node, and generate a task allocation scheme, wherein the status information includes at least one of latency, load, and bandwidth. An execution unit is configured to execute the corresponding subtasks at each edge node according to the task allocation scheme and the local data source, and generate local results; The integration unit is used to integrate the local results in the cloud to generate global analysis results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the low-latency data collaborative processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the low-latency data collaborative processing method as described in any one of claims 1 to 7.

Images