Data structured processing method and system based on structured query tree

Abstract

The present invention discloses a data structured processing method and system based on a structured query tree. The method comprises the following steps: constructing and optimizing a query tree structure to implement data query and processing in a large-scale data environment. The system dynamically adjusts the query path, identifies and optimizes bottleneck nodes and high-load nodes in the query path, extracts node features based on the structured information of the query tree, constructs a conditional probability model between nodes using a Bayesian inference algorithm, automatically identifies intrinsic relationships between nodes, optimizes the query tree execution path using an intelligent scheduling algorithm, analyzes the inference weight and load of each node, and optimizes the execution order of the query tree. The system supports cross-system data processing and meets the data transmission requirements of large-scale distributed systems by adjusting the query path and data processing strategy. The present invention effectively improves query efficiency, reduces system load, and optimizes query performance in large-scale data processing environments.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and system for data structured processing based on a structured query tree. Background Art

[0002] With the rapid development of big data technologies, cross-system data processing and query optimization have become key challenges in distributed computing systems. Especially in large-scale data environments, how to efficiently process and query massive amounts of data, improve query response speed, and reduce system load has become a pressing issue in the data processing field. Traditional query optimization methods typically rely on static query paths and fixed query strategies, lacking dynamic adaptability and targeted optimization. These methods exhibit the following significant shortcomings when faced with constantly changing data loads, network topologies, and query tasks:

[0003] 1. Lack of dynamic optimization capabilities: Existing query optimization technologies are mostly based on static models and cannot dynamically adjust query paths and execution orders based on factors such as node load and transmission delay in real time. This leads to low query efficiency, which is particularly significant in complex distributed environments.

[0004] 2. Poor query path optimization: Traditional methods typically determine the query path before query execution. This makes it difficult to deal with bottleneck nodes or resource competition that arise during the query process, and the query path cannot be adjusted in a timely manner, affecting query performance.

[0005] 3. Unintelligent resource scheduling: Existing query optimization technologies fail to fully consider the dependencies and load conditions between nodes and lack intelligent scheduling mechanisms. This leads to suboptimal resource allocation and data transmission paths, increasing the system burden.

[0006] 4. Insufficient cross-system data processing capabilities: In cross-platform or cross-system query scenarios, existing technologies are generally unable to effectively solve the problems of data format conversion and data flow between different systems, affecting data interaction and query processing between different platforms.

[0007] Therefore, how to provide a data structured processing method and system based on a structured query tree is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0008] One purpose of the present invention is to propose a data structured processing method and system based on a structured query tree. The present invention evaluates the execution performance, load, data flow and other multi-dimensional information of the query node in real time, combines with an intelligent scheduling algorithm, dynamically adjusts the query path and execution order, and optimizes the node configuration and data transmission path. It can effectively improve query efficiency, reduce system load, solve the problem of cross-system data format conversion, and has the advantages of efficient data processing capabilities, intelligent query optimization and cross-system compatibility.

[0009] The data structure processing method based on the structured query tree according to an embodiment of the present invention includes the following steps:

[0010] S1. Automatically generate a preliminary query tree framework based on the structural characteristics and metadata description information of the input data, and adaptively adjust the query tree topology, node configuration, and query path according to the storage mode, distribution characteristics, and access mode of the data source;

[0011] S2. Introducing a query tree dynamic optimization algorithm based on data dimension and query load analysis. This algorithm uses the access frequency, storage location, and processing demand information of the data source to adjust the connection mode and query path between query tree nodes in real time.

[0012] S3. Combining the adaptive conversion mechanism of data format and data structure, by analyzing the format characteristics and hierarchical relationships of nodes, dynamically designing and adjusting data format conversion rules to achieve parsing and formatting of nodes in the query tree under different task requirements;

[0013] S4. During the query tree execution process, the query path execution performance is monitored and analyzed in real time. Through dynamic evaluation of data flow and query load, the query path and execution strategy are intelligently adjusted.

[0014] S5. Adopt a cross-level data interaction mechanism based on a query tree hierarchy to automatically identify and manage cross-system data transmission requirements. Through intelligent data mapping and format conversion strategies, cross-platform data transmission and collaborative processing can be achieved.

[0015] S6. Utilizes structured information extraction from the query tree and an intelligent reasoning mechanism based on the Bayesian inference algorithm to automatically identify the inherent relationships between data, infer the conditional dependencies between multidimensional data by building a probabilistic model, and dynamically adjust the data processing order and execution strategy of relevant nodes in the query tree based on the inference results;

[0016] S7. Dynamic query path optimization technology based on query trees automatically identifies and adjusts the execution path of query trees through intelligent scheduling and optimization of query execution paths, and achieves query performance in large-scale distributed systems.

[0017] Optionally, the S1 specifically includes:

[0018] S11. Extract the structural features of the input data, analyze the hierarchical relationship, field type, and inter-field dependency information of the data, construct preliminary nodes, and assign a unique identifier to each node;

[0019] S12. Extract metadata description information of the input data to obtain the storage mode, distribution characteristics, and access mode of the data source, including the physical storage location of the data source, data update frequency, data access frequency, and its corresponding query load distribution, and associate it with the node;

[0020] S13. Based on the structured features and metadata description information of the input data, a preliminary query tree framework is constructed. The topology of the preliminary query tree is determined according to the relevance and storage location between nodes, and a preliminary node configuration and query path of the query tree are generated according to the distribution characteristics of the data source.

[0021] S14. Adjust the preliminary query tree framework and dynamically optimize the query tree topology based on the storage mode and access mode of the data source:

[0022] ;

[0023] in, Representation node and nodes The query path priority between is a node The query frequency, is a node The access delay of the function Describes the impact of storage mode and access mode on query path optimization;

[0024] S15. By configuring the nodes of the query tree, adjusting the connection mode between the nodes, determining the execution order of the nodes, and making adaptive adjustments based on the access pattern and load prediction, flexible configuration of the query path is achieved;

[0025] S16. Minimize the total cost of the query path and optimize the order of the query path by using the following objective function:

[0026] ;

[0027] in, Representative Node The query frequency, is a node To Node The data transmission load value, and Node and nodes The access delay, is the total number of nodes in the query tree;

[0028] S17. Dynamically adjust the query tree path formation rules based on the relationship between the metadata description information and the input data to adapt to different data query scenarios;

[0029] S18. Based on dynamic adjustment, generate an adaptive query tree framework that meets the characteristics of the data source and query requirements, and optimize the topology structure, node configuration and query path of the query tree.

[0030] Optionally, the S2 specifically includes:

[0031] S21. Based on the access frequency, storage location, and processing requirement information of the data source, a multidimensional feature model of the query tree nodes is constructed, including the access frequency, storage location, data size, and processing requirement data source attributes of each node;

[0032] S22. Calculate the load value of the data source node based on the multidimensional feature model:

[0033] ;

[0034] in, Indicates the The load value of each node, For nodes The frequency of visits, For nodes The storage location attribute, For nodes data processing needs, is the calculation function of the node load value;

[0035] S23. Based on the load value of the node, apply the dynamic optimization algorithm of the query tree to calculate and adjust the connection weights between the nodes in the query tree:

[0036] ;

[0037] in, Representation node and nodes The connection weights between and is the weight adjustment factor, Indicates the The load value of each node, Indicates the The load value of each node, For nodes and nodes Data transmission needs between and For nodes and nodes data processing needs;

[0038] S24. By adjusting the connection weights between nodes, the query path is optimized in real time, the connection mode of the nodes in the query tree is dynamically adjusted, and the data transmission delay and the execution efficiency of the query tree are optimized;

[0039] S25. Recalculate the query path of the query tree based on the load value and the connection weight, adjust the data transmission path between nodes and the execution order of the nodes, and optimize the topology of the query path;

[0040] S26. Minimize the total latency and data transmission load of the query path by the following objective function:

[0041] ;

[0042] in, For nodes and nodes The connection weights between is the data transmission delay between nodes, and For nodes and nodes The load value, is the total number of nodes in the query tree;

[0043] S27. By adjusting the query path and node configuration in real time, the execution order of the query tree and the query path are optimized, so that the execution delay and computing load of the query path are optimized.

[0044] Optionally, the S3 specifically includes:

[0045] S31. Extract format features of each node in the query tree, including data type, field characteristics, hierarchical relationship, and dependencies between fields, build a node formatting model based on the hierarchical relationship and association between nodes, and assign a unique formatting identifier to each node;

[0046] S32. Establish a data format conversion rule set based on the formatting requirements of the node, where each rule is dynamically generated based on the structural characteristics of the node, the amount of data, and the task requirements to meet the processing requirements of different query tasks;

[0047] S33. Define a hierarchical structure for node format conversion, analyze and determine the format conversion path between nodes based on the storage mode, distribution characteristics, and access mode of the data source, and automatically select the optimal conversion path to minimize resource consumption during the query tree execution process.

[0048] S34. Calculate the format conversion complexity of each node using the following formula:

[0049] ;

[0050] in, Indicates the The format conversion complexity of each node, For nodes The amount of data, For nodes Field-level features of For nodes Storage mode, is a function that represents the complexity of node format conversion;

[0051] S35. Based on the format conversion complexity of the nodes, dynamically optimize the data format conversion rules, adjust the format conversion order and path between nodes, and optimize the data conversion overhead and execution efficiency of each node during the query tree execution process;

[0052] S36. By dynamically adjusting the format conversion rules between nodes, the parsing and formatting processing methods of the nodes in the query tree are optimized in real time. The appropriate formatting method is automatically selected according to the task requirements, so that the data format of the node can be processed under different query tasks;

[0053] S37. By analyzing the data flow and load distribution during the query tree execution process, the format conversion strategy is adjusted in real time based on the hierarchical relationship and formatting requirements of the nodes, and the data format conversion rules are adjusted according to different query requirements during the execution process.

[0054] Optionally, the S4 specifically includes:

[0055] S41. During the query tree execution process, the query path execution performance is monitored in real time. By monitoring the data flow, processing time, access frequency, and latency of each query node, a query path performance evaluation model is constructed.

[0056] S42. By analyzing the load and data flow of each node during the query path execution process, identify the bottleneck node in the query path, determine the impact of the bottleneck node on the overall query performance, and calculate the performance index of the bottleneck node:

[0057] ;

[0058] in, Indicates the The bottleneck performance index of each node, For nodes processing time, For nodes The load value, For nodes The frequency of data flow, is the bottleneck node performance evaluation function;

[0059] S43. Based on the performance indicators of the bottleneck nodes, dynamically adjust the connection mode between nodes in the query path, reconfigure the execution order of the query path, and adjust the query path by calculating the optimization weight of each node:

[0060] ;

[0061] in, Representation node The optimization weight of and To optimize the adjustment factor, is the maximum processing time of all nodes in the query tree, is the total number of nodes in the query tree, Indicates the Bottleneck performance indicators of each node;

[0062] S44. Optimize the query tree execution order through an intelligent scheduling algorithm based on the optimization weights of the nodes, dynamically adjust the data transmission path and processing order in the query path, and optimize the latency and data transmission load in the query path;

[0063] S45. Based on the node performance data and load information collected in real time during the query tree execution process, the query tree execution strategy is adaptively adjusted, and the data processing strategy and query path of each query node are adjusted to optimize the query tree execution efficiency.

[0064] ;

[0065] in, For nodes processing load, For nodes The optimization weight of and Node The processing time and load value, is the total number of nodes in the query tree;

[0066] S46. By adjusting the query tree path and execution strategy in real time, the query path latency, load, and execution order are optimized.

[0067] Optionally, the S5 specifically includes:

[0068] S51. Based on the hierarchical structure of the query tree, by analyzing the hierarchical relationship of each node in the query tree, identify the data transmission requirements between different systems, and build a cross-system data transmission requirement model. The cross-system data transmission requirement model considers the storage mode, access method, and data type of different systems where the nodes are located, and automatically identifies the data transmission path and data transmission volume between different levels;

[0069] S52. Dynamically generate a data mapping strategy based on the cross-system data transmission requirement model, map the formats and structures of nodes at different levels in the query tree, and determine the mapping rules between each node to ensure consistency in structure and format during cross-system transmission of data;

[0070] S53. Dynamically establish cross-platform data format conversion rules based on the mapping relationship between the data transmission path and the nodes. The format conversion rules are combined with the data type, data format and query task requirements of the query tree nodes to adjust the format conversion method between each node in real time, and automatically adjust the data encoding method and storage format according to the data processing requirements of the target system.

[0071] S54. Calculate the transmission load required for cross-system data transmission using the following formula:

[0072] ;

[0073] in, Representation node With node The transmission load of cross-system data transmission requirements between For nodes With node The data transmission delay between and Node and nodes The load value, and Node and nodes Storage mode, is a computational function that transfers load across systems;

[0074] S55. Based on the cross-system data transmission load calculation results, the connection mode and data transmission path between the query tree nodes are adjusted in real time to optimize the data flow and load distribution during the cross-platform data interaction process in the query tree;

[0075] S56. Through intelligent data mapping and format conversion strategies, during the query tree execution process, the format conversion rules for cross-system data are adjusted according to the dynamic changes of the query task, so that data from different platforms can be collaboratively processed in the query tree and cross-system data interaction and transmission can be achieved;

[0076] S57. By analyzing the cross-system data interaction performance data during the query tree execution process, the data transmission path, format conversion strategy and data mapping rules are adjusted in real time to optimize the efficiency of cross-platform data transmission.

[0077] Optionally, the S6 specifically includes:

[0078] S61. Based on the structured information in the query tree, extract the characteristics of each node, including data type, hierarchical relationship, field dependency and access mode, build a relationship graph between nodes, and identify the intrinsic relationship between nodes;

[0079] S62. Use the Bayesian inference algorithm to build a conditional probability model based on the structured information between nodes:

[0080] ;

[0081] in, Represents a given parent node When the node The conditional probability of For nodes The prior probability of For a given node Time parent node The conditional probability of For the parent node The prior probability of

[0082] S63. Based on the conditional probability model, infer the dependency relationship between nodes in the query tree, calculate the conditional dependency probability between multi-dimensional nodes, identify the strength of the correlation under different query tasks, and automatically model the dependency relationship between nodes;

[0083] S64. Based on the inference results, dynamically adjust the processing order and execution strategy of the relevant nodes in the query tree, adjust the execution order of the nodes to optimize the query path, and achieve this by calculating the inference weight of the nodes:

[0084] ;

[0085] in, Representation node The inference weight of and is the weight adjustment factor, For nodes The load, For nodes Conditional probability between the parent node;

[0086] S65, according to the reasoning weight, use the greedy algorithm to optimize the scheduling of query tree nodes, by calculating the reasoning weight of each node and load , sort the nodes by priority, giving priority to nodes with higher inference weight and lower load than other nodes:

[0087] ;

[0088] in, Representation node The scheduling priority of is the inference weight of the node, is the load of the node;

[0089] S66. Adjust the execution order of the nodes in the query path according to the node order optimized by the scheduling algorithm, and optimize the execution strategy of the query tree in real time.

[0090] Optionally, the S7 specifically includes:

[0091] S71. Dynamic query path optimization technology based on the query tree, by evaluating each node in the query tree and its corresponding data processing requirements, building an execution performance model for each query node, and generating a preliminary query path topology structure;

[0092] S72. By collecting execution performance data of each query node in real time, including node load, data transmission delay, processing time, and access frequency, the query tree execution path is dynamically adjusted using an intelligent scheduling algorithm to automatically identify bottleneck nodes and high-load nodes in the query path and optimize the data transmission and processing sequence in the path.

[0093] S73. During the execution path optimization process, the execution efficiency of each query path is calculated based on the execution performance, data flow, and network bandwidth of each node, and the topology of the query path is optimized:

[0094] ;

[0095] in, Indicates the query path The execution efficiency, To query the path data traffic, and For nodes and nodes The load value, For nodes and nodes Data transmission delay between

[0096] S74. Prioritize the query paths based on the calculation results of the execution efficiency, prioritize the path with the highest execution efficiency value compared to other paths, and prioritize the path with the lowest execution efficiency value compared to other paths;

[0097] S75. During the query tree execution process, by continuously monitoring the running status of the query path, adjusting the node order in the query path in real time, and dynamically adjusting the connection method between the nodes in the query path according to the node load, to optimize the data flow and the allocation of computing resources;

[0098] S76. In combination with the characteristics of large-scale distributed systems, the execution of query paths is scheduled and optimized through distributed load balancing technology, and the node configuration and data transmission path of the query path are dynamically adjusted according to the node load, data flow and the network topology of the system.

[0099] Optionally, a data structured processing system based on a structured query tree includes the following modules:

[0100] Query tree construction module: automatically generates a query tree framework based on the input data source and its structured features, and dynamically adjusts the query tree topology according to the node type, query load, and system requirements;

[0101] Node analysis module: By introducing a data dependency analysis algorithm, it analyzes the field dependencies, hierarchical relationships, and data formats of each node in the query tree, constructs a relationship graph between nodes, and models their structural features;

[0102] Data processing module: Based on the relationship graph between nodes, it automatically builds the data processing flow, optimizes the data processing sequence according to the characteristics and dependencies of each node in the query tree, and realizes cross-node data structured processing;

[0103] Query path optimization module: Based on the node load and the multi-dimensional requirements of the query task, it collects the execution performance data of the query node in real time, dynamically adjusts the query path through an intelligent scheduling algorithm, automatically identifies bottleneck nodes, and optimizes the data transmission and processing order in the path;

[0104] Execution efficiency calculation module: Based on the query path execution efficiency calculation formula, it optimizes the query path topology structure through node load, data transmission delay and traffic data;

[0105] Query path priority sorting module: This module uses a greedy algorithm to prioritize query paths based on the query path execution efficiency calculation results, giving priority to the path with the highest execution efficiency value compared to other paths;

[0106] Data format conversion module: adjusts the conversion rules of data formats and structures based on the characteristics of query tree nodes and cross-platform data processing requirements, and realizes conversion between different data formats and cross-system collaborative processing;

[0107] System load balancing module: Combining the characteristics of large-scale distributed systems, it monitors the load conditions, data traffic and network topology of nodes in the query path through load balancing technology, and dynamically adjusts the node configuration and data transmission path in the query path.

[0108] The beneficial effects of the present invention are:

[0109] (1) The present invention solves the problems of low processing efficiency and high response delay of traditional methods in large-scale data environments by using dynamic query path optimization technology based on structured query trees, combining real-time evaluation of query node execution performance data, and adopting intelligent scheduling algorithms to dynamically adjust the query path. The present invention can identify bottleneck nodes and high-load nodes in the query path in real time, and by optimizing the topological structure and execution order of the query path, greatly improve the execution efficiency of the query tree and the response speed of the system, thus meeting the query requirements in high-concurrency and high-load data environments.

[0110] (2) The present invention solves the problem of data format differences and processing capability mismatches between different systems by constructing a cross-system data processing model. By dynamically analyzing the format characteristics, data transmission requirements, and data types of each node in the query tree, the data format conversion rules are designed and adjusted, so that data can be seamlessly transmitted and processed between different systems. The present invention also automatically adjusts the cross-system data transmission path according to the execution requirements of the query tree, optimizes data flow and load distribution, ensures the efficiency, compatibility, and consistency of the query path between different systems, and greatly reduces the performance bottleneck caused by data format incompatibility or transmission delay.

[0111] (3) The present invention supports dynamic expansion and flexible adjustment of the query tree structure, and can make intelligent adjustments in real time based on the performance data of the query execution path. By continuously monitoring the node sequence and data flow in the query path, the connection mode and processing order between nodes can be adjusted in a timely manner to achieve optimal data transmission and computing resource allocation. The present invention can automatically adjust the query path node configuration according to the load changes and network topology structure of the large-scale distributed system, greatly improving the scalability and flexibility of the query tree in a distributed environment, effectively reducing query latency, and ensuring the efficient operation of the system in a large-scale data processing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0113] Figure 1 This is the overall framework diagram of the data structured processing method and system based on the structured query tree proposed by the present invention;

[0114] Figure 2 This is a working diagram of the cross-system data processing model proposed in the present invention. DETAILED DESCRIPTION

[0115] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0116] refer to Figure 1-2 , a data structured processing method and system based on a structured query tree, comprising the following steps:

[0117] S1. Automatically generate a preliminary query tree framework based on the structural characteristics and metadata description information of the input data, and adaptively adjust the query tree topology, node configuration, and query path according to the storage mode, distribution characteristics, and access mode of the data source;

[0118] S2. Introducing a query tree dynamic optimization algorithm based on data dimension and query load analysis. This algorithm uses the access frequency, storage location, and processing demand information of the data source to adjust the connection mode and query path between query tree nodes in real time.

[0119] S3. Combining the adaptive conversion mechanism of data format and data structure, by analyzing the format characteristics and hierarchical relationships of nodes, dynamically designing and adjusting data format conversion rules to achieve parsing and formatting of nodes in the query tree under different task requirements;

[0120] S4. During the query tree execution process, the query path execution performance is monitored and analyzed in real time. Through dynamic evaluation of data flow and query load, the query path and execution strategy are intelligently adjusted.

[0121] S5. Adopt a cross-level data interaction mechanism based on a query tree hierarchy to automatically identify and manage cross-system data transmission requirements. Through intelligent data mapping and format conversion strategies, cross-platform data transmission and collaborative processing can be achieved.

[0122] S6. Utilizes structured information extraction from the query tree and an intelligent reasoning mechanism based on the Bayesian inference algorithm to automatically identify the inherent relationships between data, infer the conditional dependencies between multidimensional data by building a probabilistic model, and dynamically adjust the data processing order and execution strategy of relevant nodes in the query tree;

[0123] S7. Dynamic query path optimization technology based on query trees automatically identifies and adjusts the execution path of query trees through intelligent scheduling and optimization of query execution paths, and achieves query performance in large-scale distributed systems.

[0124] In this embodiment, S1 specifically includes:

[0125] S11. Extract the structural features of the input data, analyze the hierarchical relationship, field type, and inter-field dependency information of the data, construct preliminary nodes, and assign a unique identifier to each node;

[0126] S12. Extract metadata description information of the input data to obtain the storage mode, distribution characteristics, and access mode of the data source, including the physical storage location of the data source, data update frequency, data access frequency, and its corresponding query load distribution, and associate it with the node;

[0127] S13. Based on the structured features and metadata description information of the input data, a preliminary query tree framework is constructed. The topology of the preliminary query tree is determined according to the relevance and storage location between nodes, and a preliminary node configuration and query path of the query tree are generated according to the distribution characteristics of the data source.

[0128] S14. Adjust the preliminary query tree framework and dynamically optimize the query tree topology based on the storage mode and access mode of the data source:

[0129] ;

[0130] in, Representation node and nodes The query path priority between is a node The query frequency, is a node The access delay of the function Describes the impact of storage mode and access mode on query path optimization;

[0131] S15. By configuring the nodes of the query tree, adjusting the connection mode between the nodes, determining the execution order of the nodes, and making adaptive adjustments based on the access pattern and load prediction, flexible configuration of the query path is achieved;

[0132] S16. Minimize the total cost of the query path and optimize the order of the query path by using the following objective function:

[0133] ;

[0134] in, Representative Node The query frequency, is a node To Node The data transmission load value, and Node and nodes The access delay, is the total number of nodes in the query tree;

[0135] S17. Dynamically adjust the query tree path formation rules based on the relationship between the metadata description information and the input data to adapt to different data query scenarios;

[0136] S18. Based on dynamic adjustment, generate an adaptive query tree framework that meets the characteristics of the data source and query requirements, and optimize the topology structure, node configuration and query path of the query tree.

[0137] This implementation builds an adaptive query tree framework, combining the structured characteristics of input data with metadata descriptors to achieve dynamic optimization and adaptive adjustment of query paths. By analyzing data hierarchies, field dependencies, and access patterns, this approach effectively optimizes the query tree topology, reduces query costs, and improves query efficiency, ensuring efficient response to diverse query requirements and data source characteristics in large-scale data environments.

[0138] In this embodiment, S2 specifically includes:

[0139] S21. Based on the access frequency, storage location, and processing requirement information of the data source, a multidimensional feature model of the query tree nodes is constructed, including the access frequency, storage location, data size, and processing requirement data source attributes of each node;

[0140] S22. Calculate the load value of the data source node based on the multidimensional feature model:

[0141] ;

[0142] in, Indicates the The load value of each node, For nodes The frequency of visits, For nodes The storage location attribute, For nodes data processing needs, is the calculation function of the node load value;

[0143] S23. Based on the load value of the node, apply the dynamic optimization algorithm of the query tree to calculate and adjust the connection weights between the nodes in the query tree:

[0144] ;

[0145] in, Representation node and nodes The connection weights between and is the weight adjustment factor, Indicates the The load value of each node, Indicates the The load value of each node, For nodes and nodes Data transmission needs between and For nodes and nodes data processing needs;

[0146] S24. By adjusting the connection weights between nodes, the query path is optimized in real time, the connection mode of the nodes in the query tree is dynamically adjusted, and the data transmission delay and the execution efficiency of the query tree are optimized;

[0147] S25. Recalculate the query path of the query tree based on the load value and the connection weight, adjust the data transmission path between nodes and the execution order of the nodes, and optimize the topology of the query path;

[0148] S26. Minimize the total latency and data transmission load of the query path by the following objective function:

[0149] ;

[0150] in, For nodes and nodes The connection weights between is the data transmission delay between nodes, and For nodes and nodes The load value, is the total number of nodes in the query tree;

[0151] S27. By adjusting the query path and node configuration in real time, the execution order of the query tree and the query path are optimized, so that the execution delay and computing load of the query path are optimized.

[0152] This implementation dynamically optimizes the query tree's node configuration and query paths by building a multidimensional feature model of query tree nodes based on access frequency, storage location, and processing requirements. By accurately calculating node load values and inter-node connection weights, this optimizes query paths, reduces data transmission delays, and improves query efficiency.

[0153] In this embodiment, S3 specifically includes:

[0154] S31. Extract format features of each node in the query tree, including data type, field characteristics, hierarchical relationship, and dependencies between fields, build a node formatting model based on the hierarchical relationship and association between nodes, and assign a unique formatting identifier to each node;

[0155] S32. Establish a data format conversion rule set based on the formatting requirements of the node, where each rule is dynamically generated based on the structural characteristics of the node, the amount of data, and the task requirements to meet the processing requirements of different query tasks;

[0156] S33. Define a hierarchical structure for node format conversion, analyze and determine the format conversion path between nodes based on the storage mode, distribution characteristics, and access mode of the data source, and automatically select the optimal conversion path to minimize resource consumption during the query tree execution process.

[0157] S34. Calculate the format conversion complexity of each node using the following formula:

[0158] ;

[0159] in, Indicates the The format conversion complexity of each node, For nodes The amount of data, For nodes Field-level features of For nodes Storage mode, is a function that represents the complexity of node format conversion;

[0160] S35. Based on the format conversion complexity of the nodes, dynamically optimize the data format conversion rules, adjust the format conversion order and path between nodes, and optimize the data conversion overhead and execution efficiency of each node during the query tree execution process;

[0161] S36. By dynamically adjusting the format conversion rules between nodes, the parsing and formatting processing methods of the nodes in the query tree are optimized in real time. The appropriate formatting method is automatically selected according to the task requirements, so that the data format of the node can be processed under different query tasks;

[0162] S37. By analyzing the data flow and load distribution during the query tree execution process, the format conversion strategy is adjusted in real time based on the hierarchical relationship and formatting requirements of the nodes, and the data format conversion rules are adjusted according to different query requirements during the execution process.

[0163] This implementation constructs a node formatting model, extracts format feature information for each node in the query tree, and dynamically generates data format conversion rules that adapt to different query tasks. By analyzing the structural characteristics, data volume, and task requirements of the nodes, the data format conversion path and sequence are optimized to minimize resource consumption and improve query tree execution efficiency.

[0164] In this embodiment, the S4 specifically includes:

[0165] S41. During the query tree execution process, the query path execution performance is monitored in real time. By monitoring the data flow, processing time, access frequency, and latency of each query node, a query path performance evaluation model is constructed.

[0166] S42. By analyzing the load and data flow of each node during the query path execution process, identify the bottleneck node in the query path, determine the impact of the bottleneck node on the overall query performance, and calculate the performance index of the bottleneck node:

[0167] ;

[0168] in, Indicates the The bottleneck performance index of each node, For nodes processing time, For nodes The load value, For nodes The frequency of data flow, is the bottleneck node performance evaluation function;

[0169] S43. Based on the performance indicators of the bottleneck nodes, dynamically adjust the connection mode between nodes in the query path, reconfigure the execution order of the query path, and adjust the query path by calculating the optimization weight of each node:

[0170] ;

[0171] in, Representation node The optimization weight of and To optimize the adjustment factor, is the maximum processing time of all nodes in the query tree, is the total number of nodes in the query tree, Indicates the Bottleneck performance indicators of each node;

[0172] S44. Optimize the query tree execution order through an intelligent scheduling algorithm based on the optimization weights of the nodes, dynamically adjust the data transmission path and processing order in the query path, and optimize the latency and data transmission load in the query path;

[0173] S45. Based on the node performance data and load information collected in real time during the query tree execution process, the query tree execution strategy is adaptively adjusted, and the data processing strategy and query path of each query node are adjusted to optimize the query tree execution efficiency.

[0174] ;

[0175] in, For nodes processing load, For nodes The optimization weight of and Node The processing time and load value, is the total number of nodes in the query tree;

[0176] S46. By adjusting the query tree path and execution strategy in real time, the query path latency, load, and execution order are optimized.

[0177] This implementation monitors node performance in real time during query tree execution, constructs a performance evaluation model for the query path, identifies bottleneck nodes, and calculates their performance metrics, thereby dynamically optimizing the query tree's execution order. By adjusting node optimization weights and execution paths, data flow and processing order are optimized, reducing latency and data transmission load.

[0178] In this embodiment, the S5 specifically includes:

[0179] S51. Based on the hierarchical structure of the query tree, by analyzing the hierarchical relationship of each node in the query tree, identify the data transmission requirements between different systems, and build a cross-system data transmission requirement model. The cross-system data transmission requirement model considers the storage mode, access method, and data type of different systems where the nodes are located, and automatically identifies the data transmission path and data transmission volume between different levels;

[0180] S52. Dynamically generate a data mapping strategy based on the cross-system data transmission requirement model, map the formats and structures of nodes at different levels in the query tree, and determine the mapping rules between each node to ensure consistency in structure and format during cross-system transmission of data;

[0181] S53. Dynamically establish cross-platform data format conversion rules based on the mapping relationship between the data transmission path and the nodes. The format conversion rules are combined with the data type, data format and query task requirements of the query tree nodes to adjust the format conversion method between each node in real time, and automatically adjust the data encoding method and storage format according to the data processing requirements of the target system.

[0182] S54. Calculate the transmission load required for cross-system data transmission using the following formula:

[0183] ;

[0184] in, Representation node With node The transmission load of cross-system data transmission requirements between For nodes With node The data transmission delay between and Node and nodes The load value, and Node and nodes Storage mode, is a computational function that transfers load across systems;

[0185] S55. Based on the cross-system data transmission load calculation results, the connection mode and data transmission path between the query tree nodes are adjusted in real time to optimize the data flow and load distribution during the cross-platform data interaction process in the query tree;

[0186] S56. Through intelligent data mapping and format conversion strategies, during the query tree execution process, the format conversion rules for cross-system data are adjusted according to the dynamic changes of the query task, so that data from different platforms can be collaboratively processed in the query tree and cross-system data interaction and transmission can be achieved;

[0187] S57. By analyzing the cross-system data interaction performance data during the query tree execution process, the data transmission path, format conversion strategy and data mapping rules are adjusted in real time to optimize the efficiency of cross-platform data transmission.

[0188] This implementation method constructs a cross-system data transmission demand model, analyzes the hierarchical relationship and data transmission requirements of nodes in the query tree, automatically identifies the data transmission path and data volume between different systems, ensures the consistency of data during cross-system transmission, and combines intelligent data mapping and format conversion strategies to dynamically adjust data format, encoding method and storage form, optimize resource consumption during data transmission, and further optimize the efficiency of cross-system data transmission by real-time analysis of data interaction performance, thereby improving data interaction and transmission efficiency during query tree execution.

[0189] In this embodiment, S6 specifically includes:

[0190] S61. Based on the structured information in the query tree, extract the characteristics of each node, including data type, hierarchical relationship, field dependency and access mode, build a relationship graph between nodes, and identify the intrinsic relationship between nodes;

[0191] S62. Use the Bayesian inference algorithm to build a conditional probability model based on the structured information between nodes:

[0192] ;

[0193] in, Represents a given parent node When the node The conditional probability of For nodes The prior probability of For a given node Time parent node The conditional probability of For the parent node The prior probability of

[0194] S63. Based on the conditional probability model, infer the dependency relationship between nodes in the query tree, calculate the conditional dependency probability between multi-dimensional nodes, identify the strength of the correlation under different query tasks, and automatically model the dependency relationship between nodes;

[0195] S64. Based on the inference results, dynamically adjust the processing order and execution strategy of the relevant nodes in the query tree, adjust the execution order of the nodes to optimize the query path, and achieve this by calculating the inference weight of the nodes:

[0196] ;

[0197] in, Representation node The inference weight of and is the weight adjustment factor, For nodes The load, For nodes Conditional probability between the parent node;

[0198] S65, according to the reasoning weight, use the greedy algorithm to optimize the scheduling of query tree nodes, by calculating the reasoning weight of each node and load , sort the nodes by priority, giving priority to nodes with higher inference weight and lower load than other nodes:

[0199] ;

[0200] in, Representation node The scheduling priority of is the inference weight of the node, is the load of the node;

[0201] S66. Adjust the execution order of the nodes in the query path according to the node order optimized by the scheduling algorithm, and optimize the execution strategy of the query tree in real time.

[0202] This implementation combines the Bayesian reasoning algorithm and the greedy algorithm, and through dynamic reasoning and optimization of the execution order of nodes in the query tree, automatically identifies the dependencies and conditional probabilities between nodes, optimizes the processing order and execution strategy of the query path, and efficiently adjusts the scheduling priority of the nodes. It can significantly improve the execution efficiency of the query tree under different query tasks, reduce query latency, optimize system load, and provide an efficient query optimization method.

[0203] In this embodiment, the S7 specifically includes:

[0204] S71. Dynamic query path optimization technology based on the query tree, by evaluating each node in the query tree and its corresponding data processing requirements, building an execution performance model for each query node, and generating a preliminary query path topology structure;

[0205] S72. By collecting execution performance data of each query node in real time, including node load, data transmission delay, processing time, and access frequency, the query tree execution path is dynamically adjusted using an intelligent scheduling algorithm to automatically identify bottleneck nodes and high-load nodes in the query path and optimize the data transmission and processing sequence in the path.

[0206] S73. During the execution path optimization process, the execution efficiency of each query path is calculated based on the execution performance, data flow, and network bandwidth of each node, and the topology of the query path is optimized:

[0207] ;

[0208] in, Indicates the query path The execution efficiency, To query the path data traffic, and For nodes and nodes The load value, For nodes and nodes Data transmission delay between

[0209] S74. Prioritize the query paths based on the calculation results of the execution efficiency, prioritize the path with the highest execution efficiency value compared to other paths, and prioritize the path with the lowest execution efficiency value compared to other paths;

[0210] S75. During the query tree execution process, by continuously monitoring the running status of the query path, adjusting the node order in the query path in real time, and dynamically adjusting the connection method between the nodes in the query path according to the node load, to optimize the data flow and the allocation of computing resources;

[0211] S76. In combination with the characteristics of large-scale distributed systems, the execution of query paths is scheduled and optimized through distributed load balancing technology, and the node configuration and data transmission path of the query path are dynamically adjusted according to the node load, data flow and the network topology of the system.

[0212] This implementation method uses dynamic query path optimization technology, combined with execution performance models and intelligent scheduling algorithms, to automatically identify and optimize bottleneck nodes and high-load nodes in the query path, adjust the node order and data transmission path in real time, and based on distributed load balancing technology, significantly improve the execution efficiency of the query path, reduce data transmission delays and computing loads, and ensure the efficiency and scalability of query operations in large-scale distributed systems.

[0213] In this embodiment, the data structured processing system based on the structured query tree includes the following modules:

[0214] Query tree construction module: automatically generates a query tree framework based on the input data source and its structured features, and dynamically adjusts the query tree topology according to the node type, query load, and system requirements;

[0215] Node analysis module: By introducing a data dependency analysis algorithm, it analyzes the field dependencies, hierarchical relationships, and data formats of each node in the query tree, constructs a relationship graph between nodes, and models their structural features;

[0216] Data processing module: Based on the relationship graph between nodes, it automatically builds the data processing flow, optimizes the data processing sequence according to the characteristics and dependencies of each node in the query tree, and realizes cross-node data structured processing;

[0217] Query path optimization module: Based on the node load and the multi-dimensional requirements of the query task, it collects the execution performance data of the query node in real time, dynamically adjusts the query path through an intelligent scheduling algorithm, automatically identifies bottleneck nodes, and optimizes the data transmission and processing order in the path;

[0218] Execution efficiency calculation module: Based on the query path execution efficiency calculation formula, it optimizes the query path topology structure through node load, data transmission delay and traffic data;

[0219] Query path priority sorting module: This module uses a greedy algorithm to prioritize query paths based on the query path execution efficiency calculation results, giving priority to the path with the highest execution efficiency value compared to other paths;

[0220] Data format conversion module: adjusts the conversion rules of data formats and structures based on the characteristics of query tree nodes and cross-platform data processing requirements, and realizes conversion between different data formats and cross-system collaborative processing;

[0221] System load balancing module: Combining the characteristics of large-scale distributed systems, it monitors the load conditions, data traffic and network topology of nodes in the query path through load balancing technology, and dynamically adjusts the node configuration and data transmission path in the query path.

[0222] Example 1:

[0223] In order to verify the actual application effect of the present invention in query tree optimization and dynamic adjustment of execution paths, the present invention was applied to the query system of a large e-commerce platform. The platform processes millions of query requests per day, and the system involves multi-level data storage, complex query conditions and cross-system data interaction. Against the background of a rapid increase in data volume, there has been a significant delay in the platform's query response time, which has seriously affected the user experience and the platform's service quality. The traditional static query path optimization method can no longer adapt to the rapidly changing system load and data requirements. In order to solve the above problems, the platform decided to introduce the dynamic query path optimization technology provided by the present invention to improve the performance and response efficiency of the query system.

[0224] In the query system of the platform, query requests are processed in the form of a query tree. Each node in the query tree represents a query operation or data processing task, and the nodes are connected through query paths, forming a complex dependency relationship. Due to the diverse business types of the e-commerce platform, the execution efficiency of the query path directly determines the length of the query response time. However, due to the wide variety of query requests, complex data types, and frequent load fluctuations, the original query optimization method is difficult to effectively cope with. In order to improve query efficiency, the platform adopts the dynamic query path optimization technology of the present invention, which dynamically adjusts the execution order of the query tree and the connection method between nodes by real-time monitoring of the query node load, data transmission delay, processing time and access frequency.

[0225] During the implementation process, the platform first generates a preliminary query tree topology by analyzing the structured information of each node, including data types, field dependencies, and access patterns, during the query tree construction phase. Based on this information, the system automatically evaluates the execution performance of each node, including its load, processing time, and data transmission latency. By collecting execution data from each node in real time, the platform uses an intelligent scheduling algorithm to dynamically adjust query paths. Specifically, the system calculates the execution efficiency of each query path, prioritizes paths with higher execution efficiency, and adjusts the node order in the query path based on the real-time execution status to optimize query response time.

[0226] In order to verify the effectiveness of this technology, the platform compared the query response time and system load before and after the introduction of the technology of the present invention. Before implementation, the platform's query requests often had response times of more than 10 seconds during peak hours, resulting in the loss of some users. After introducing the dynamic optimization algorithm of the present invention, the system can dynamically adjust the query path according to the execution efficiency of the node, successfully shortening the query response time to less than 5 seconds. In addition, the platform has greatly improved the system's concurrent processing capabilities, reduced the load on the query path, and alleviated the pressure on the server by adjusting the node order and load distribution in the query path. The following is a data table comparing the query response time and system load before and after implementation:

[0227] Table 1 Comparison of query response time and system load

[0228]

[0229] As shown in Table 1, the introduction of our dynamic query path optimization technology significantly reduced query response time from 10.5 seconds to 4.8 seconds, and the query success rate increased from 92% to 98%. System load also decreased significantly, with CPU utilization dropping from 85% to 60%, demonstrating that query optimization resulted in a more balanced load distribution and more efficient query processing.

[0230] To further validate the effectiveness of this technology, the platform compared query request volume and query performance data before and after implementation. During peak query request periods, the system dynamically adjusted its query path optimization algorithm to identify and adjust bottlenecks and high-load nodes in the query path in real time, thereby avoiding slow responses and processing failures caused by overloaded query systems. The following are the platform's query performance data during peak query periods:

[0231] Table 2 Comparison of performance data during query peak period

[0232]

[0233] As shown in Table 2, during peak query request volumes, the platform's query system, after introducing the technology presented in this paper, successfully increased its query request processing capacity from 1,500 to 2,200 per second. Query response time was significantly reduced from 12.4 seconds to 5.2 seconds. System stability was also greatly improved, with the number of response failures or timeouts dropping from 45 to just 5. The system's CPU utilization was effectively controlled, dropping from 95% to 72%, demonstrating that the system can maintain efficient and stable query services even under high load.

[0234] Through the above-mentioned embodiments, the platform successfully solved the query system's query latency and load issues caused by increased data volume and query path complexity. Dynamic query path optimization technology, by adjusting query paths and node order in real time, enables the system to efficiently process large-scale query requests and effectively allocate computing resources, improving query response speed and system stability. Ultimately, after introducing the technology of this invention, the platform's query system significantly improved its processing power and query efficiency, making the system more efficient and reliable, and greatly improving the user experience.

[0235] In summary, the dynamic query path optimization technology provided by the present invention can significantly improve the performance of large-scale data query systems. While ensuring query efficiency, it effectively reduces system load and improves query response speed and stability. It is suitable for large-scale distributed systems such as e-commerce and finance that have high query performance requirements.

[0236] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A data structuring processing method based on a structured query tree, characterized in that: The steps include: S1. Based on the structured features and metadata description information of the input data, the preliminary query tree framework is automatically generated, and the topology, node configuration and query path of the query tree are adaptively adjusted; S2. Introducing a query tree dynamic optimization algorithm based on data dimension and query load analysis. This algorithm uses the access frequency, storage location, and processing demand information of the data source to adjust the connection mode and query path between query tree nodes in real time. S3, combining the adaptive conversion mechanism of data format and data structure, dynamically designing and adjusting data format conversion rules by analyzing the format characteristics and hierarchical relationships of nodes; S4. During the query tree execution process, the query path execution performance is monitored and analyzed in real time. Through dynamic evaluation of data flow and query load, the query path and execution strategy are intelligently adjusted. S5. Adopt a cross-level data interaction mechanism based on a query tree hierarchy to automatically identify and manage cross-system data transmission requirements. Through intelligent data mapping and format conversion strategies, cross-platform data transmission and collaborative processing can be achieved. S6. Utilizes structured information extraction from the query tree and an intelligent reasoning mechanism based on the Bayesian inference algorithm to automatically identify the inherent relationships between data, infer the conditional dependencies between multidimensional data by building a probabilistic model, and dynamically adjust the data processing order and execution strategy of related nodes in the query tree; S7, dynamic query path optimization technology based on query trees, automatically identifies and adjusts the execution path of query trees through intelligent scheduling and optimization of query execution paths; The S7 specifically includes: S71. Dynamic query path optimization technology based on the query tree, by evaluating each node in the query tree and its corresponding data processing requirements, building an execution performance model for each query node, and generating a preliminary query path topology structure; S72. By collecting execution performance data of each query node in real time, including node load, data transmission delay, processing time, and access frequency, the query tree execution path is dynamically adjusted using an intelligent scheduling algorithm to automatically identify bottleneck nodes and high-load nodes in the query path and optimize the data transmission and processing sequence in the path. S73. During the execution path optimization process, the execution efficiency of each query path is calculated based on the execution performance, data flow, and network bandwidth of each node, and the topology of the query path is optimized: ； in, Indicates the query path The execution efficiency, To query the path data traffic, and For nodes and nodes The load value, For nodes and nodes Data transmission delay between S74. Prioritize the query paths based on the calculation results of the execution efficiency, prioritize the path with the highest execution efficiency value compared to other paths, and prioritize the path with the lowest execution efficiency value compared to other paths; S75. During the query tree execution process, by continuously monitoring the running status of the query path, adjusting the node order in the query path in real time, and dynamically adjusting the connection method between the nodes in the query path according to the node load, to optimize the data flow and the allocation of computing resources; S76. In combination with the characteristics of large-scale distributed systems, the execution of query paths is scheduled and optimized through distributed load balancing technology, and the node configuration and data transmission path of the query path are dynamically adjusted according to the node load, data flow and the network topology of the system.

2. The data structured processing method based on the structured query tree according to claim 1, characterized in that: Said S1 specifically includes: S11. Extract the structural features of the input data, analyze the hierarchical relationship, field type, and inter-field dependency information of the data, construct preliminary nodes, and assign a unique identifier to each node; S12. Extract metadata description information of the input data to obtain the storage mode, distribution characteristics, and access mode of the data source, including the physical storage location of the data source, data update frequency, data access frequency, and its corresponding query load distribution, and associate it with the node; S13. Based on the structured features and metadata description information of the input data, a preliminary query tree framework is constructed. The topology of the preliminary query tree is determined according to the relevance and storage location between nodes, and a preliminary node configuration and query path of the query tree are generated according to the distribution characteristics of the data source. S14. Adjust the preliminary query tree framework and dynamically optimize the query tree topology based on the storage mode and access mode of the data source: ； in, Representation node and nodes The query path priority between is a node The query frequency, is a node The access delay of the function Describes the impact of storage mode and access mode on query path optimization; S15. By configuring the nodes of the query tree, adjusting the connection mode between the nodes, determining the execution order of the nodes, and making adaptive adjustments based on the access pattern and load prediction, flexible configuration of the query path is achieved; S16. Minimize the total cost of the query path and optimize the order of the query path by using the following objective function: ； in, Representative Node The query frequency, is a node To Node The data transmission load value, and Node and nodes The access delay, is the total number of nodes in the query tree; S17. Dynamically adjust the query tree path formation rules based on the relationship between the metadata description information and the input data to adapt to different data query scenarios; S18. Based on dynamic adjustment, generate an adaptive query tree framework that meets the characteristics of the data source and query requirements, and optimize the topology structure, node configuration and query path of the query tree.

3. The data structured processing method based on the structured query tree according to claim 2, characterized in that: The S2 specifically includes: S21. Based on the access frequency, storage location, and processing requirement information of the data source, a multidimensional feature model of the query tree nodes is constructed, including the access frequency, storage location, data size, and processing requirement data source attributes of each node; S22. Calculate the load value of the data source node based on the multidimensional feature model: ； in, Indicates the The load value of each node, For nodes The frequency of visits, For nodes The storage location attribute, For nodes data processing needs, is the calculation function of the node load value; S23. Based on the load value of the node, apply the dynamic optimization algorithm of the query tree to calculate and adjust the connection weights between the nodes in the query tree: ； in, Representation node and nodes The connection weights between and is the weight adjustment factor, Indicates the The load value of each node, Indicates the The load value of each node, For nodes and nodes Data transmission needs between and For nodes and nodes data processing needs; S24. By adjusting the connection weights between nodes, the query path is optimized in real time, the connection mode of the nodes in the query tree is dynamically adjusted, and the data transmission delay and the execution efficiency of the query tree are optimized; S25. Recalculate the query path of the query tree based on the load value and the connection weight, adjust the data transmission path between nodes and the execution order of the nodes, and optimize the topology of the query path; S26. Minimize the total latency and data transmission load of the query path by the following objective function: ； in, For nodes and nodes The connection weights between is the data transmission delay between nodes, and For nodes and nodes The load value, is the total number of nodes in the query tree; S27. By adjusting the query path and node configuration in real time, the execution order of the query tree and the query path are optimized, so that the execution delay and computing load of the query path are optimized.

4. The data structure processing method based on the structured query tree according to claim 1, characterized in that: The S3 specifically includes: S31. Extract format features of each node in the query tree, including data type, field characteristics, hierarchical relationship, and dependencies between fields, build a node formatting model based on the hierarchical relationship and association between nodes, and assign a unique formatting identifier to each node; S32. Establish a data format conversion rule set based on the formatting requirements of the node, where each rule is dynamically generated based on the structural characteristics of the node, the amount of data, and the task requirements to meet the processing requirements of different query tasks; S33. Define a hierarchical structure for node format conversion, analyze and determine the format conversion path between nodes based on the storage mode, distribution characteristics, and access mode of the data source, and automatically select the optimal conversion path to minimize resource consumption during the query tree execution process. S34. Calculate the format conversion complexity of each node using the following formula: ； in, Indicates the The format conversion complexity of each node, For nodes The amount of data, For nodes Field-level features of For nodes Storage mode, is a function that represents the complexity of node format conversion; S35. Based on the format conversion complexity of the nodes, dynamically optimize the data format conversion rules, adjust the format conversion order and path between nodes, and optimize the data conversion overhead and execution efficiency of each node during the query tree execution process; S36. By dynamically adjusting the format conversion rules between nodes, the parsing and formatting processing methods of the nodes in the query tree are optimized in real time. The appropriate formatting method is automatically selected according to the task requirements, so that the data format of the node can be processed under different query tasks; S37. By analyzing the data flow and load distribution during the query tree execution process, the format conversion strategy is adjusted in real time based on the hierarchical relationship and formatting requirements of the nodes, and the data format conversion rules are adjusted according to different query requirements during the execution process.

5. The data structure processing method based on the structured query tree according to claim 1, characterized in that: The S4 specifically includes: S41. During the query tree execution process, the query path execution performance is monitored in real time. By monitoring the data flow, processing time, access frequency, and latency of each query node, a query path performance evaluation model is constructed. S42. By analyzing the load and data flow of each node during the query path execution process, identify the bottleneck node in the query path, determine the impact of the bottleneck node on the overall query performance, and calculate the performance index of the bottleneck node: ； in, Indicates the The bottleneck performance index of each node, For nodes processing time, For nodes The load value, For nodes The frequency of data flow, is the bottleneck node performance evaluation function; S43. Based on the performance indicators of the bottleneck nodes, dynamically adjust the connection mode between nodes in the query path, reconfigure the execution order of the query path, and adjust the query path by calculating the optimization weight of each node: ； in, Representation node The optimization weight of and To optimize the adjustment factor, is the maximum processing time of all nodes in the query tree, is the total number of nodes in the query tree, Indicates the Bottleneck performance indicators of each node; S44. Optimize the query tree execution order through an intelligent scheduling algorithm based on the optimization weights of the nodes, dynamically adjust the data transmission path and processing order in the query path, and optimize the latency and data transmission load in the query path; S45. Based on the node performance data and load information collected in real time during the query tree execution process, the query tree execution strategy is adaptively adjusted, and the data processing strategy and query path of each query node are adjusted to optimize the query tree execution efficiency. ； in, For nodes processing load, For nodes The optimization weight of and Node The processing time and load value, is the total number of nodes in the query tree; S46. By adjusting the query tree path and execution strategy in real time, the query path latency, load, and execution order are optimized.

6. The data structure processing method based on the structured query tree according to claim 1, characterized in that: The S5 specifically includes: S51. Based on the hierarchical structure of the query tree, by analyzing the hierarchical relationship of each node in the query tree, identify the data transmission requirements between different systems, and build a cross-system data transmission requirement model. The cross-system data transmission requirement model considers the storage mode, access method, and data type of different systems where the nodes are located, and automatically identifies the data transmission path and data transmission volume between different levels; S52. Dynamically generate a data mapping strategy based on the cross-system data transmission requirement model, map the formats and structures of nodes at different levels in the query tree, and determine the mapping rules between each node to ensure consistency in structure and format during cross-system transmission of data; S53. Dynamically establish cross-platform data format conversion rules based on the mapping relationship between the data transmission path and the nodes. The format conversion rules are combined with the data type, data format and query task requirements of the query tree nodes to adjust the format conversion method between each node in real time, and automatically adjust the data encoding method and storage format according to the data processing requirements of the target system. S54. Calculate the transmission load required for cross-system data transmission using the following formula: ； in, Representation node With node The transmission load of cross-system data transmission requirements between For nodes With node The data transmission delay between and Node and nodes The load value, and Node and nodes Storage mode, is a computational function that transfers load across systems; S55. Based on the cross-system data transmission load calculation results, the connection mode and data transmission path between the query tree nodes are adjusted in real time to optimize the data flow and load distribution during the cross-platform data interaction process in the query tree; S56. Through intelligent data mapping and format conversion strategies, during the query tree execution process, the format conversion rules for cross-system data are adjusted according to the dynamic changes of the query task, so that data from different platforms can be collaboratively processed in the query tree and cross-system data interaction and transmission can be achieved; S57. By analyzing the cross-system data interaction performance data during the query tree execution process, the data transmission path, format conversion strategy and data mapping rules are adjusted in real time to optimize the efficiency of cross-platform data transmission.

7. The data structured processing method based on the structured query tree according to claim 1, characterized in that: The S6 specifically includes: S61. Based on the structured information in the query tree, extract the characteristics of each node, including data type, hierarchical relationship, field dependency and access mode, build a relationship graph between nodes, and identify the intrinsic relationship between nodes; S62. Use the Bayesian inference algorithm to build a conditional probability model based on the structured information between nodes: ； in, Represents a given parent node When the node The conditional probability of For nodes The prior probability of For a given node Time parent node The conditional probability of For the parent node The prior probability of S63. Based on the conditional probability model, infer the dependency relationship between nodes in the query tree, calculate the conditional dependency probability between multi-dimensional nodes, identify the strength of the correlation under different query tasks, and automatically model the dependency relationship between nodes; S64. Based on the inference results, dynamically adjust the processing order and execution strategy of the relevant nodes in the query tree, adjust the execution order of the nodes to optimize the query path, and achieve this by calculating the inference weight of the nodes: ； in, Representation node The inference weight of and is the weight adjustment factor, For nodes The load, For nodes Conditional probability between the parent node; S65, according to the reasoning weight, use the greedy algorithm to optimize the scheduling of query tree nodes, by calculating the reasoning weight of each node and load , sort the nodes by priority, giving priority to nodes with higher inference weight and lower load than other nodes: ； in, Representation node The scheduling priority of is the inference weight of the node, is the load of the node; S66. Adjust the execution order of the nodes in the query path according to the node order optimized by the scheduling algorithm, and optimize the execution strategy of the query tree in real time.

8. A data structured processing system based on a structured query tree, applied to the data structured processing method based on a structured query tree according to any one of claims 1 to 7, characterized in that: Includes the following modules: Query tree construction module: automatically generates a query tree framework based on the input data source and its structured features, and dynamically adjusts the query tree topology according to the node type, query load, and system requirements; Node analysis module: By introducing a data dependency analysis algorithm, it analyzes the field dependencies, hierarchical relationships, and data formats of each node in the query tree, constructs a relationship graph between nodes, and models their structural features; Data processing module: Based on the relationship graph between nodes, automatically build the data processing flow and optimize the data processing order according to the characteristics and dependency relationships of each node in the query tree; Query path optimization module: Based on the node load and the multi-dimensional requirements of the query task, it collects the execution performance data of the query node in real time, dynamically adjusts the query path through an intelligent scheduling algorithm, automatically identifies bottleneck nodes, and optimizes the data transmission and processing order in the path; Execution efficiency calculation module: Based on the query path execution efficiency calculation formula, it optimizes the query path topology structure through node load, data transmission delay and traffic data; Query path priority sorting module: This module uses a greedy algorithm to prioritize query paths based on the query path execution efficiency calculation results, giving priority to the path with the highest execution efficiency value compared to other paths; Data format conversion module: adjusts the conversion rules of data format and structure according to the characteristics of query tree nodes and cross-platform data processing requirements; System load balancing module: Combining the characteristics of large-scale distributed systems, it monitors the load conditions, data traffic and network topology of nodes in the query path through load balancing technology, and dynamically adjusts the node configuration and data transmission path in the query path.

Images