Data processing system and method for water conservancy heterogeneous big data components

Abstract

The present invention provides a data processing system and method for water conservancy heterogeneous big data components, which relates to the field of big data technology. The system includes: a processing module, which is used to process data lineage for multiple big data components in the same process to obtain data processing tasks; a structure acquisition module, which is used to convert data processing tasks into data in SQL format, and preprocess the data in SQL format to obtain preprocessed data; a structure conversion module, which is used to generate an abstract syntax tree for the preprocessed data, and convert the abstract syntax tree into a parse tree to generate a query solution; a filtering module, which is used to filter the query solution to obtain a filtered query solution, and evaluate the filtered query solution to obtain an evaluation result. The present invention adopts a universal data lineage processing method to realize the processing of data lineage for multiple big data components in the same process.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing system and method for water conservancy heterogeneous big data components. Background Art

[0002] With the gradual advancement of information construction in the field of water conservancy, there are more and more multi-source heterogeneous big data processing needs. These data are collected from different channels and platforms, with different structures and large scales, which are quite challenging to process. In the traditional data processing process, different processing strategies and methods are usually required for different big data components. This practice has increased the complexity of data processing and affected the efficiency and accuracy of processing to a certain extent. At the same time, due to the lack of unified data lineage management methods, it is sometimes difficult to clearly track the source and flow of data, and it is impossible to fully understand the various changes in the data during the processing process, which undoubtedly affects the credibility of the data and its potential value. Furthermore, task failures may occasionally occur during data processing. Unfortunately, due to the lack of effective failure response strategies and data recovery mechanisms, task failures often mean data loss or damage, which undoubtedly brings certain troubles to water conservancy work. At present, there are still some obstacles to data integration and sharing between different data sources, which limits the maximum utilization of data resources in the field of water conservancy. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a data processing system and method for water conservancy heterogeneous big data components, adopt a universal data lineage processing method, and realize the data lineage processing of multiple big data components with the same process.

[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0005] In a first aspect, a data processing system for water conservancy heterogeneous big data components, the system comprises:

[0006] The processing module is used to process the data lineage of various big data components in the same process to obtain data processing tasks;

[0007] A structure acquisition module is used to convert the data processing task into data in SQL format and preprocess the data in SQL format to obtain preprocessed data;

[0008] A structure conversion module, used to generate an abstract syntax tree from the preprocessed data, and convert the abstract syntax tree into a parse tree to generate a query solution;

[0009] A filtering module, used for filtering query solutions to obtain filtered query solutions, and evaluating the filtered query solutions to obtain evaluation results;

[0010] The lineage construction module is used to obtain the data item dependencies and expressions of the query solution based on the evaluation results, and extract the sub-modules of the lineage information to construct data lineage information.

[0011] Furthermore, the data lineage of multiple big data components is processed in the same process to obtain data processing tasks, including:

[0012] Get the environmental status data of the big data component;

[0013] Based on the environmental status data of the big data component, Calculated environmental status assessment index ,in, Indicates the number of verified valid data entries; Indicates the total number of data entries in the dataset; represents the weight coefficient; Indicates The value of a key indicator; Indicates the number of indicators; Indicates the number of data entries that meet the criteria; , and represents the weight coefficient; Represents an index, used to traverse all key indicators;

[0014] According to historical data, Calculate the threshold of the environmental status evaluation index when the big data component is operating normally ,According to the environmental status evaluation index and threshold, the normal environmental status of the big data component is determined, where, It represents the mean value of the environmental status assessment index in historical data; represents the standard deviation of the environmental status assessment index; Represents the total number of environmental status assessment indices in the historical data set; Represents an index, which is used to traverse each data point in the historical data set; Indicates The value of the historical environmental status assessment index; , and represents weight;

[0015] According to the normal environmental status of the big data component, the data lineage is processed with the same process to obtain the data processing task.

[0016] Further, the data processing task is converted into SQL format, and the data in SQL format is preprocessed to obtain preprocessed data, including:

[0017] Parse the data processing tasks and obtain the instruction set of the big data component;

[0018] According to the instruction set of the big data component, read the data and convert the data processing tasks into data in SQL format;

[0019] Perform data cleansing on the SQL formatted data to obtain cleaned data;

[0020] The cleaned data is passed through Perform filtering to obtain filtered data, wherein: It's at the time The filtered data, is the cleaned data sequence at time point The data value of It is at every point in time; is the index;

[0021] The filtered data is normalized to obtain preprocessed data.

[0022] Furthermore, the preprocessed data is used to generate an abstract syntax tree, and the abstract syntax tree is converted into a parse tree to generate a query solution, including:

[0023] Scan the preprocessed data using a scanner, and decompose the preprocessed data into a token sequence, where each token is an atomic element in the preprocessed data, including a keyword, an identifier, an operator, or a value;

[0024] The token sequence is converted into an abstract syntax tree through a recursive descent parser. In the abstract syntax tree, each node represents an instance of a grammatical structure, and the node type reflects the type of the grammatical structure.

[0025] Take the abstract syntax tree as an individual in the initial population and define a fitness function for evaluating the quality of each abstract syntax tree individual corresponding to the query solution;

[0026] According to the fitness function, the corresponding abstract syntax tree individual is selected as the parent, two parent abstract syntax tree individuals are randomly selected, and a new abstract syntax tree individual is generated by exchanging subtrees or nodes; the selected abstract syntax tree individual is randomly modified, and the child abstract syntax tree individual generated by the genetic operation is converted into a query plan, and the query plan is continuously optimized through multiple rounds of genetic iterations until a preset number of iterations is reached to obtain the final abstract syntax tree individual;

[0027] Convert the final abstract syntax tree into a specific query plan.

[0028] Furthermore, the calculation formula of the fitness function is:

[0029] ;

[0030] in, The weight representing the execution efficiency; Indicates the number of records returned by error; The weight representing the accuracy; , and represents weight; , , , , , , and They represent the maximum query execution time, actual execution time, number of correct records, total number of records, number of error records, number of join operations, number of subqueries, and number of nested levels.

[0031] Furthermore, the token sequence is converted into an abstract syntax tree through a recursive descent parser, including:

[0032] Starting from the beginning of the token sequence, a recursive descent parser is used for parsing; the parser reads and matches tokens one by one according to the grammar rules, and gradually builds each node of the abstract syntax tree;

[0033] When a complete grammatical structure is matched, a new abstract syntax tree node is created and its type is set;

[0034] For each newly created node, if it contains substructures, recursively call the parser to process the substructures and add the substructures as child nodes to the current node. This process continues until the token sequence is completely consumed or an unmatched token is encountered.

[0035] After the abstract syntax tree is constructed, a post-processing phase is started, starting from the root node of the abstract syntax tree, and performing a depth-first traversal. During the traversal, for each accessed node, a metadata query operation is performed according to its type and content. The query operation involves searching the symbol table and accessing the database.

[0036] If valid metadata is found, assign it to the metadata field of the current abstract syntax tree node. If the abstract syntax tree node does not require metadata or the metadata query fails, keep the empty value or set the default value.

[0037] When the entire abstract syntax tree is traversed and all required metadata are queried and assigned, an abstract syntax tree containing complete structure and metadata information is output.

[0038] Further, the query solutions are filtered to obtain filtered query solutions, and the filtered query solutions are evaluated to obtain evaluation results, including:

[0039] Read all pending query plans and store them in a processing queue;

[0040] Call the syntax parser to perform syntax checks on the query plans in the processing queue one by one, mark and remove those query plans that do not meet the syntax specifications, so that the remaining query plans are grammatically correct;

[0041] Perform logical verification on query plans that pass the syntax check, identify query plans that are logically unreasonable or have potential errors, and remove query plans with logical problems from the processing queue;

[0042] Hash the remaining query solutions and generate a unique hash value for each query solution;

[0043] Compare hash values ​​to find and remove duplicate query solutions so that each query solution is unique in the processing queue;

[0044] Automatically parse the table names, field names and associated relationships in the remaining query solutions; build an internal data model based on the table names, field names and associated relationships in the remaining query solutions;

[0045] Analyze the internal data model to obtain the dependencies and flow paths between data;

[0046] Based on the dependencies and flow paths between data, a data lineage graph is constructed to show the association and flow between data;

[0047] Based on the data lineage graph and data volume, the performance of each query solution is evaluated to obtain the evaluation results.

[0048] Furthermore, based on the data lineage graph and data volume, each query solution is evaluated for performance to obtain evaluation results, including:

[0049] Based on the data lineage graph and data volume, each query solution is Conduct performance evaluation to obtain evaluation results;

[0050] in, represents the performance evaluation value; , , and represents the weight coefficient; Indicates the number of data entries; Indicates the average size of each record; Indicates the number of dependencies; Indicates The data volume weight of each dependency; Indicates The amount of data associated with each dependency; Indicates the total number of dependencies; Indicates The coupling weight of each dependency; Indicates Dependencies related to The connection strength of each dependency; Indicates The processing time weight of each dependency; Indicates processing The time required for each dependency; ,in, is the number of direct connections, indicating dependencies and dependencies The number of direct relationships between is the frequency of data interaction, indicating the dependency and dependencies The frequency of data exchange between them; Representing dependencies The weight of Represents an index.

[0051] Furthermore, according to the evaluation results, the data item dependencies and expressions of the query solution are obtained, and the submodules of the lineage information are extracted to construct the data lineage information, including:

[0052] According to the evaluation results, determine the initial input data item set and the output data item set;

[0053] Analyze the data flow information provided in the evaluation results and establish direct mapping relationships between data items;

[0054] Using the direct mapping relationship, starting from the output data item, reversely search for all input data items that affect its generation, and starting from the input data item, forward search for all output data items derived from it;

[0055] For each relevant data item found, a Cartesian product operation is performed to obtain a Cartesian product operation result;

[0056] Based on the results of the Cartesian product operation, complete data lineage information is generated, which depicts the dependencies and transformation paths of all data items from the original input to the final output.

[0057] In a second aspect, a data processing method for water conservancy heterogeneous big data components is provided, the method comprising:

[0058] Process the data lineage of multiple big data components in the same process to obtain data processing tasks;

[0059] Convert the data processing task into data in SQL format, and preprocess the data in SQL format to obtain preprocessed data;

[0060] Generate an abstract syntax tree from the preprocessed data, and convert the abstract syntax tree into a parse tree to generate a query solution;

[0061] Filtering the query solutions to obtain filtered query solutions, and evaluating the filtered query solutions to obtain evaluation results;

[0062] According to the evaluation results, the data item dependencies and expressions of the query solution are obtained, and the sub-modules of the lineage information are extracted to construct the data lineage information.

[0063] The above solution of the present invention includes at least the following beneficial effects:

[0064] By designing a universal data lineage processing method, it is possible to process data lineage for multiple big data components in the same process, simplifying the data processing process and improving the efficiency and accuracy of data processing; data lineage records the link relationship in the entire life cycle of data from generation to extinction. Through this system, it is easy to track the source and destination of data and understand the changes in data during the processing process, thereby enhancing the traceability of data. By parsing the query plan of data processing tasks and constructing data lineage information, it can help users better understand the data dependencies in the data processing process. During the data processing process, tasks may fail due to various reasons. Through this system, users can promptly discover and handle these failed tasks to avoid data loss or damage. At the same time, due to the existence of data lineage information, users can easily restore failed tasks and continue the data processing process, improving the fault tolerance of the system. By associating data lineage information with data items in multi-source data integration applications, the system realizes data integration and sharing between different data sources, making it easier for the water conservancy field to obtain and utilize data resources from different channels. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of a data processing system for water conservancy heterogeneous big data components provided by an embodiment of the present invention.

[0066] Figure 2 It is a schematic diagram of a data processing method for water conservancy heterogeneous big data components provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in a form and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0068] like Figure 1 As shown, an embodiment of the present invention proposes a data processing system for water conservancy heterogeneous big data components, the system comprising:

[0069] The processing module 11 is used to process data lineage of various big data components in the same process to obtain data processing tasks;

[0070] The structure acquisition module 12 is used to convert the data processing task into data in SQL format and pre-process the data in SQL format to obtain pre-processed data;

[0071] A structure conversion module 13, used to generate an abstract syntax tree from the preprocessed data, and convert the abstract syntax tree into a parse tree to generate a query solution;

[0072] A filtering module 14 is used to filter the query solutions to obtain filtered query solutions, and evaluate the filtered query solutions to obtain evaluation results;

[0073] The lineage construction module 15 is used to obtain the data item dependencies and expressions of the query solution according to the evaluation results, and extract the sub-modules of the lineage information to construct the data lineage information.

[0074] In the embodiment of the present invention, the same process is used for data lineage processing of multiple big data components, which helps to standardize the operation process and reduce the maintenance and learning costs. Due to the use of a unified processing process, the system can more easily integrate new big data components, thereby improving the compatibility and scalability of the system. Converting data processing tasks into SQL format realizes the standardization of data format. By preprocessing the data in SQL format, errors and inconsistencies in the data can be cleaned and corrected, thereby improving data quality. Generating an abstract syntax tree from the preprocessed data and further converting it into a parse tree helps to generate a more efficient query solution, thereby improving query efficiency. Through the conversion of the abstract syntax tree and the parse tree, the system can handle complex query requirements more flexibly. By filtering and evaluating query solutions, those solutions with grammatical errors, unreasonable logic or low performance can be eliminated, thereby improving the quality of the overall query solution. Evaluating the query solution helps to reasonably allocate system resources and avoid unnecessary waste of resources. By constructing data lineage information, the source and flow path of the data can be clearly traced, thereby improving the traceability of the data. Data lineage information helps to identify potential data risks and problem points, so as to take corresponding risk management measures.

[0075] In a preferred embodiment of the present invention, multiple big data components are processed according to data lineage in the same process to obtain data processing tasks, including:

[0076] Get the environmental status data of the big data component;

[0077] Based on the environmental status data of the big data component, Calculated environmental status assessment index ,in, Indicates the number of verified valid data entries; Indicates the total number of data entries in the dataset; represents the weight coefficient; Indicates The value of a key indicator; Indicates the number of indicators; Indicates the number of data entries that meet the criteria; , and represents the weight coefficient; Represents an index, used to traverse all key indicators;

[0078] According to historical data, Calculate the threshold of the environmental status evaluation index when the big data component is operating normally ,According to the environmental status evaluation index and threshold, the normal environmental status of the big data component is determined, where, It represents the mean value of the environmental status assessment index in historical data; represents the standard deviation of the environmental status assessment index; Represents the total number of environmental status assessment indices in the historical data set; Represents an index, which is used to traverse each data point in the historical data set; Indicates The value of the historical environmental status assessment index; , and represents weight;

[0079] According to the normal environmental status of the big data component, the data lineage is processed with the same process to obtain the data processing task.

[0080] In an embodiment of the present invention, by obtaining the environmental status data of the big data component, the system can monitor the operating status of the component in real time and detect abnormal situations in a timely manner. By calculating the environmental status evaluation index, the data quality of the big data component can be quantitatively evaluated, and valid data entries and data entries that meet the standards can be identified. According to the environmental status evaluation index, system resources can be reasonably allocated, and components with poor performance or low data quality can be optimized or replaced. By calculating the threshold of the environmental status evaluation index during normal operation, the system can automatically identify and distinguish between normal and abnormal environmental states. Based on the normal environmental state of the big data component, data lineage processing is performed in the same process to simplify the data processing process and reduce the complexity of operation. Through a complete data lineage processing process, data governance capabilities are strengthened.

[0081] In a specific embodiment of the present invention, various environmental status data are collected and obtained when the big data component is running. The data may include CPU usage, memory usage, disk I / O, network bandwidth usage, task execution time, task success rate, etc. The environmental status evaluation index is calculated by a formula that comprehensively considers multiple factors, including the number of verified valid data entries, the total number of data entries in the data set, the value of the key indicator and its weight coefficient, and the number of data entries that meet the standards. Specifically: Indicates the percentage of valid data, reflecting the accuracy and completeness of the data. It represents the weighted sum of all key indicators, which may include indicators of performance, stability, availability, etc. It represents the percentage of data that meets the standards, reflecting the quality of the data. α, β and γ are weight coefficients used to adjust the relative importance of the above parts in the final evaluation index. Through this formula, we can get an index that comprehensively evaluates the environmental status of big data components.

[0082] A threshold is calculated based on historical data to determine whether the environmental status of the big data component is normal. The calculation of the threshold takes into account the mean, standard deviation, and higher-order statistics (such as skewness and kurtosis) of the environmental status assessment index in the historical data. Specifically, the mean represents the average level of the environmental status assessment index in the historical data, and the standard deviation represents the fluctuation of the environmental status assessment index in the historical data. The subsequent part involves the adjustment of skewness (measurement of the asymmetry of data distribution) and kurtosis (measurement of the sharpness of data distribution). , and is a weight coefficient used to adjust the relative importance of the above parts in the final threshold calculation. Through this formula, a threshold can be obtained. Compare the current environmental status assessment index with the threshold calculated in the previous step. If the environmental status assessment index is within a reasonable range of the threshold, it can be considered that the big data component is in a normal environmental state. Conversely, if the environmental status assessment index exceeds this range, it may mean that the big data component has performance problems or potential failures. After confirming that the big data component is in a normal environmental state, the data lineage is processed according to the same process. Data lineage refers to all the transformations and flow paths that data passes through from its original source to its final destination. By tracking and analyzing data lineage, you can understand the source and destination of the data and how it changes during the entire processing process.

[0083] In a preferred embodiment of the present invention, the data processing task is converted into SQL format, and the data in SQL format is preprocessed to obtain preprocessed data, including:

[0084] Parse the data processing tasks and obtain the instruction set of the big data component;

[0085] According to the instruction set of the big data component, read the data and convert the data processing tasks into data in SQL format;

[0086] Perform data cleansing on the SQL formatted data to obtain cleaned data;

[0087] The cleaned data is passed through Perform filtering to obtain filtered data, wherein: It's at the time The filtered data, is the cleaned data sequence at time point The data value of It is at every point in time; is the index;

[0088] The filtered data is normalized to obtain preprocessed data.

[0089] In an embodiment of the present invention, by parsing the data processing task and obtaining the instruction set of the big data component, the task objectives and processing flow can be clarified, thereby improving the efficiency of data processing. By converting the data processing task into data in SQL format, the powerful functions of the SQL language can be used to perform efficient data query and operation. The data cleaning step can remove noise, outliers and duplicate data in the original data, thereby improving the quality and accuracy of the data. The filtering process can further smooth the data, remove unnecessary fluctuations and interference, and make the data more stable and reliable. The data normalization step converts data of different scales or units into a unified range, enhancing the comparability and availability of the data. By decomposing the data processing task into multiple clear steps, the complexity of the overall processing can be reduced. Cleaning and filtering the data can reduce the impact of abnormal data on the system and improve the robustness and stability of the system. Through flexible instruction set acquisition and SQL data conversion, the system can support diverse data processing needs.

[0090] In a specific embodiment of the present invention, it is necessary to perform detailed screening, sorting, aggregation and analysis on the data processing tasks to obtain the data processing tasks, and convert the data processing tasks into an instruction set that can be executed by the big data component (such as Apache Hadoop, Apache Spark). The instruction set is usually an API call or script command specific to the big data processing framework, which is used to instruct the big data component on how to operate the data. Using the instruction set of the big data component, the original data is read from the storage system through data retrieval and streaming technology, so that the data can be loaded into the processing system efficiently and accurately. Since the big data processing framework usually supports SQL-like query language, the data processing tasks can be more easily expressed through SQL statements, so the original data processing tasks are converted into equivalent SQL queries. In the SQL environment, the WHERE clause is used to filter out data that does not meet specific conditions, or the UPDATE statement is used to correct erroneous data values, which is used to correct, delete or replace inaccurate, incomplete, malformed or duplicate data to obtain cleaned data; the cleaned data is filtered, and for each time point t, the data is initialized. 0, and calculate the initial value of the sum of weights (i.e., the denominator part), traverse all time points from t-N to t+N (i.e., for each k value), and obtain the corresponding cleaned data value x t +k, for each x t + k, calculate its weighted value and add it to At the same time, the weight of each data point is added to the sum of the weights. Finally, Divide by the sum of the weights to get the filtered data value at time point t. After filtering, the noise and outliers in the original data will be smoothed, making the data sequence smoother and more stable. Scale the filtered data to a uniform range (usually between 0 and 1) and find the minimum value in the filtered data sequence and maximum value , for each data point in the data series ,use After normalization, each data point will be converted to a value between 0 and 1. , which is the normalized data value.

[0091] In a preferred embodiment of the present invention, the preprocessed data is used to generate an abstract syntax tree, and the abstract syntax tree is converted into a parse tree to generate a query solution, including:

[0092] Scan the preprocessed data using a scanner, and decompose the preprocessed data into a token sequence, where each token is an atomic element in the preprocessed data, including a keyword, an identifier, an operator, or a value;

[0093] The token sequence is converted into an abstract syntax tree through a recursive descent parser. In the abstract syntax tree, each node represents an instance of a grammatical structure, and the node type reflects the type of the grammatical structure.

[0094] Take the abstract syntax tree as an individual in the initial population and define a fitness function for evaluating the quality of each abstract syntax tree individual corresponding to the query solution;

[0095] According to the fitness function, the corresponding abstract syntax tree individual is selected as the parent, two parent abstract syntax tree individuals are randomly selected, and a new abstract syntax tree individual is generated by exchanging subtrees or nodes; the selected abstract syntax tree individual is randomly modified, and the child abstract syntax tree individual generated by the genetic operation is converted into a query plan, and the query plan is continuously optimized through multiple rounds of genetic iterations until a preset number of iterations is reached to obtain the final abstract syntax tree individual;

[0096] Convert the final abstract syntax tree into a specific query plan.

[0097] In an embodiment of the present invention, the pre-processed data is decomposed into a token sequence by a scanner, so that each element in the data, including keywords and identifiers, can be more accurately identified and parsed. The recursive descent parser converts the token sequence into an abstract syntax tree, which improves the accuracy and efficiency of syntax parsing. As an intermediate representation of the query solution, the abstract syntax tree can more clearly reflect the logic and structure of the query. By defining a fitness function to evaluate the pros and cons of each abstract syntax tree individual, the query solution can be selected and optimized more targeted. In the genetic iteration process, new abstract syntax tree individuals are generated by operations such as exchanging subtrees or nodes and random modifications, which can continuously explore and optimize the space of query solutions. The final abstract syntax tree individuals obtained after multiple rounds of genetic iterations often correspond to more efficient and better-performing query solutions. Using the abstract syntax tree as an intermediate representation enables the system to more easily process and parse complex query requirements, and the application of genetic algorithms also enables the system to adaptively generate and optimize query solutions when facing different data and query requirements. Through the automated scanning, parsing and optimization process, the workload of manually writing and optimizing query solutions is reduced, and the labor cost is reduced.

[0098] In a specific embodiment of the present invention, the scanner is a component of a compiler or an interpreter, which is used to read the input character sequence and decompose it into a series of tokens, wherein a token is an atomic element in the input data, which represents a basic unit in the language, such as a keyword, an identifier, an operator or a value. The scanner starts reading from the starting position of the input data, and according to the grammatical rules of the language, the extracted tokens are put into a sequence to form a token sequence, wherein the grammatical rules of the language are the conventional rules of highly abstract combination and aggregation relationships in the language, including part of speech collocation, word order arrangement, etc. A recursive descent parser is a top-down parser that converts a token sequence into an abstract syntax tree (AST) according to the grammatical rules of the language. The abstract syntax tree (AST) is a tree representation of the abstract syntax structure of the source code, and each node on the tree represents a structure in the source code. The parser starts from the beginning of the token sequence and tries to match and consume tokens in the token sequence according to the production rules of the language. Whenever a match is successful, the parser creates a corresponding AST node and connects it to the constructed AST part. This process is performed recursively until the entire token sequence is consumed and a complete AST is generated.

[0099] In a genetic algorithm, a population is a set of multiple individuals, each of which represents a possible solution to a problem, where each individual corresponds to an AST; the fitness function is a function used to evaluate the quality of each individual in the population. The above-generated AST is used as an individual in the initial population, and the fitness function is calculated to quantitatively evaluate the quality of the query solution corresponding to each AST individual. According to the value of the fitness function, excellent individuals are selected from the current population as parents to generate the next generation. Two parent individuals are randomly selected, and new individuals are generated by exchanging some structures between them (such as subtrees or nodes). Random changes are made to the selected individuals, such as replacing a node, deleting or adding subtrees, etc.; the diversity of the population is increased to prevent premature convergence to the local optimal solution, and the newly generated individuals are evaluated: the newly generated individuals are evaluated by applying the fitness function to determine their quality. In each round of iteration, excellent individuals are selected according to the fitness function, and crossover and mutation operations are performed on these individuals to generate new individuals, evaluate the newly generated individuals, and add them to the next generation population. Repeat the above steps until the preset number of iterations is reached. After the genetic algorithm terminates, the AST individual with the highest fitness is selected. Then, based on the structure and information of this AST, it is converted back into a specific query solution (such as a SQL query statement).

[0100] In a preferred embodiment of the present invention, the calculation formula of the fitness function is:

[0101] ;

[0102] in, The weight representing the execution efficiency; Indicates the number of records returned by error; The weight representing the accuracy; , and represents weight; , , , , , , and They represent the maximum query execution time, actual execution time, number of correct records, total number of records, number of error records, number of join operations, number of subqueries, and number of nested levels.

[0103] In an embodiment of the present invention, by considering the maximum time and actual execution time of the query execution, the system can be motivated to find faster query paths and methods, thereby improving the overall execution efficiency. The consideration of accuracy weights and the number of records returned in error helps to reduce errors in query results and improve the accuracy of queries. Accurate query results can avoid wrong decisions caused by misleading information. By weighing the number of connection operations, the number of subqueries, and the number of nested levels, the complexity of the query can be controlled to avoid unnecessary consumption of system resources due to overly complex queries. The setting of multiple weights in the evaluation function allows the system to flexibly adjust the priorities of different indicators according to actual needs. By incorporating multiple key performance indicators into a comprehensive evaluation function, the system can automatically optimize and adjust itself according to these indicators.

[0104] In a preferred embodiment of the present invention, a token sequence is converted into an abstract syntax tree by a recursive descent parser, comprising:

[0105] Starting from the beginning of the token sequence, a recursive descent parser is used for parsing; the parser reads and matches tokens one by one according to the grammar rules, and gradually builds each node of the abstract syntax tree;

[0106] When a complete grammatical structure is matched, a new abstract syntax tree node is created and its type is set;

[0107] For each newly created node, if it contains substructures, recursively call the parser to process the substructures and add the substructures as child nodes to the current node. This process continues until the token sequence is completely consumed or an unmatched token is encountered.

[0108] After the abstract syntax tree is constructed, a post-processing phase is started, starting from the root node of the abstract syntax tree, and performing a depth-first traversal. During the traversal, for each accessed node, a metadata query operation is performed according to its type and content. The query operation involves searching the symbol table and accessing the database.

[0109] If valid metadata is found, assign it to the metadata field of the current abstract syntax tree node. If the abstract syntax tree node does not require metadata or the metadata query fails, keep the empty value or set the default value.

[0110] When the entire abstract syntax tree is traversed and all required metadata are queried and assigned, an abstract syntax tree containing complete structure and metadata information is output.

[0111] In an embodiment of the present invention, the recursive descent parser can perform token matching and grammatical structure parsing in strict accordance with predefined grammatical rules, thereby ensuring that the generated abstract syntax tree accurately reflects the grammatical structure and intent of the original data. As a structured data representation form, the abstract syntax tree can clearly show the internal logic and components of complex queries or instructions. By recursively constructing an abstract syntax tree, the system can process queries containing multiple layers of nesting and complex logical structures. The depth-first traversal and metadata query operations in the post-processing stage can efficiently assign the required metadata to the abstract syntax tree nodes, avoiding frequent database access or symbol table lookups during the parsing process, thereby improving the overall processing efficiency. By adding metadata fields to the abstract syntax tree nodes and querying and assigning valid metadata, each node not only contains structural information, but also enriches semantic information. In the process of constructing the abstract syntax tree, when encountering unmatched tokens, the system can adopt corresponding error handling mechanisms, such as reporting errors, ignoring or providing default values, etc., thereby enhancing the robustness and fault tolerance of the system. The abstract syntax tree with a complete structure and rich metadata can be used as input for advanced functions such as code generation and automated query optimization, thereby improving the automation level and processing efficiency of the system.

[0112] In a specific embodiment of the present invention, a recursive descent parser is initialized and a token sequence to be parsed is prepared, wherein the token sequence is decomposed by a scanner from preprocessed data and contains atomic elements such as keywords, identifiers, operators and numerical values. Starting from the starting position of the token sequence, the recursive descent parser reads and matches tokens one by one according to predefined grammatical rules. The parser gradually builds each node of the AST by identifying different token types and patterns. When the parser matches a complete grammatical structure, a new AST node is created and the type of the node is set to reflect its grammatical structure. For each newly created AST node, if it contains substructures, the parser will recursively call itself to process these substructures. The result of the recursive call is that the substructure is parsed into a child node and added to the child node list of the current node until the token sequence is completely consumed or an unmatched token is encountered. When all tokens have been processed and no unmatched tokens are encountered, the construction process of the AST is completed, and an AST representing the grammatical structure of the input data is obtained. After the AST is built, a post-processing stage is entered. Starting from the root node of the AST, perform a depth-first traversal. For each accessed AST node, perform the corresponding metadata query operation according to its type and content. If the metadata query operation successfully returns valid metadata, assign these metadata to the metadata field of the current AST node. If an AST node does not require metadata, or the metadata query fails, keep the metadata field of the node empty or set a default value. When the entire AST is traversed and all required metadata are queried and assigned, an AST containing complete structure and metadata information is obtained.

[0113] In a preferred embodiment of the present invention, filtering query solutions to obtain filtered query solutions, and evaluating the filtered query solutions to obtain evaluation results include:

[0114] Read all pending query plans and store them in a processing queue;

[0115] Call the syntax parser to perform syntax checks on the query plans in the processing queue one by one, mark and remove those query plans that do not meet the syntax specifications, so that the remaining query plans are grammatically correct;

[0116] Perform logical verification on query plans that pass the syntax check, identify query plans that are logically unreasonable or have potential errors, and remove query plans with logical problems from the processing queue;

[0117] Hash the remaining query solutions and generate a unique hash value for each query solution;

[0118] Compare hash values ​​to find and remove duplicate query solutions so that each query solution is unique in the processing queue;

[0119] Automatically parse the table names, field names and associated relationships in the remaining query solutions; build an internal data model based on the table names, field names and associated relationships in the remaining query solutions;

[0120] Analyze the internal data model to obtain the dependencies and flow paths between data;

[0121] Based on the dependencies and flow paths between data, a data lineage graph is constructed to show the association and flow between data;

[0122] Based on the data lineage graph and data volume, the performance of each query solution is evaluated to obtain the evaluation results.

[0123] In an embodiment of the present invention, through syntax checking and logic verification, it is possible to ensure that the query solutions in the processing queue are grammatically correct and logically reasonable and error-free, thereby improving the quality of the query solutions and reducing the possibility of execution errors and exceptions. The query solutions are hashed and compared to effectively remove duplicate query solutions and avoid unnecessary repeated processing. By automatically parsing the table names, field names and associations in the query solutions and building an internal data model, the structure and connection between the data are clearer and easier to understand. The construction of a data lineage graph intuitively displays the associations and flow paths between the data, optimizing data flow and management. Performance evaluation of each query solution based on the data lineage graph and data volume helps to optimize the query strategy. Through automated and intelligent processing flows, the need for manual intervention and manual inspection is reduced, thereby reducing the maintenance cost of the system.

[0124] In a specific embodiment of the present invention, the system accesses a predetermined storage location and reads all query solutions to be processed therefrom. The query solutions exist in various forms, such as SQL statements and query scripts. After reading, the system puts these query solutions into a processing queue. The processing queue is a data structure used to manage the query solutions to be processed in a first-in-first-out order. The system calls a syntax parser, wherein the parser understands and verifies the syntax structure of the query solution. The syntax parser takes out the query solutions in the processing queue one by one and performs a syntax check on them. If the syntax of a query solution does not meet the specification, the syntax parser will mark the solution and remove it from the processing queue. In this way, the remaining query solutions in the processing queue are all grammatically correct. In addition to grammatical correctness, the system will further perform logical verification on the query solutions that pass the syntax check to identify those query solutions that are logically unreasonable or have potential errors. For example, a query solution may require querying a non-existent field, or use an incorrect association condition when connecting multiple tables. If it is found that the query solution has a logical problem, the system will remove it from the processing queue. Then the remaining query solutions in the processing queue are all grammatically correct and logically reasonable. The system will perform hash processing on each remaining query plan in the processing queue. Hashing is a method of mapping data of any length to data of fixed length. Each query plan is converted into a unique hash value, which is the unique identifier of the query plan. The system will compare the hash values ​​of all query plans to find duplicate query plans. If the hash values ​​of two or more query plans are the same, it means that they are duplicates. The system will retain one of the query plans and remove the remaining duplicate plans to ensure that each query plan in the processing queue is unique and avoid processing the same query plan multiple times. For each remaining query plan in the processing queue, the system will automatically parse the table name, field name, and the relationship between tables.

[0125] Based on the parsed table names, field names and associations, the system will build an internal data model, which is an abstract representation of the actual data structure in the database and reflects the relationship between the data tables and fields involved in the query plan. By analyzing the internal data model, the system can determine the dependencies between data tables and the data flow path. Dependencies refer to the associations and dependencies between data tables, such as which tables are the data source or target of other tables; the flow path describes how data flows from one table to another, which helps to understand the execution process of the query plan and the flow of data. Based on the analyzed dependencies and flow paths, the system will build a data lineage graph. The data lineage graph is a visual representation that intuitively shows the association and flow between data. In this graph, nodes usually represent data tables or fields, while edges represent the dependencies and flow relationships between data. Finally, the system will calculate the number of nodes in the data lineage graph, including data tables, fields, and conversion processes. The more nodes there are, the more complex the data elements involved in the query. At the same time, different types of nodes will also affect the complexity. Edges represent data flow or dependency in the data lineage graph. The more edges there are, the higher the correlation between the data, and the more complex the query may be. If there are loops or nested structures in the data lineage graph, this usually means that there is recursive or iterative processing in the query, which will increase the complexity and execution time of the query. The system checks the size of the data tables involved, including the number of rows, columns, and bytes stored. In addition to the current amount of data, the system also considers the growth trend of the data. If the data grows at a faster rate, the query performance may decline over time; the distribution of the data (such as uniform distribution, skewed distribution, etc.) and sparsity (the proportion of null values ​​or zero values ​​in the data) also affect the query performance. For example, sparse data may require more storage space and processing time. Based on the above complexity and data volume evaluation, the system estimates key indicators such as query execution time and resource consumption based on historical data to obtain evaluation results.

[0126] In a preferred embodiment of the present invention, based on the data lineage graph and the data volume, a performance evaluation is performed on each query solution to obtain an evaluation result, including:

[0127] Based on the data lineage graph and data volume, each query solution is Conduct performance evaluation to obtain evaluation results;

[0128] in, represents the performance evaluation value; , , and represents the weight coefficient; Indicates the number of data entries; Indicates the average size of each record; Indicates the number of dependencies; Indicates The data volume weight of each dependency; Indicates The amount of data associated with each dependency; Indicates the total number of dependencies; Indicates The coupling weight of each dependency; Indicates Dependencies related to The connection strength of each dependency; Indicates The processing time weight of each dependency; Indicates processing The time required for each dependency; ,in, is the number of direct connections, indicating dependencies and dependencies The number of direct relationships between is the frequency of data interaction, indicating the dependency and dependencies The frequency of data exchange between them; Representing dependencies The weight of Represents an index.

[0129] In an embodiment of the present invention, by comprehensively considering multiple dimensions such as the number of data entries, record size, number of dependencies and their related data volume, degree of coupling and processing time, a comprehensive and in-depth query solution performance analysis can be provided, which helps to discover potential performance bottlenecks and optimization points. Combined with the dependencies and flow paths in the data lineage graph, as well as the specific data volume, degree of coupling and processing time of each dependency, the execution efficiency of the query solution can be accurately predicted. Understanding the data volume, processing time and connection strength of each dependency helps the system to allocate computing, storage and network resources more reasonably. By analyzing the frequency of data interaction and the connection strength between dependencies, dependencies that frequently exchange data can be identified, which helps to optimize the data flow path and reduce unnecessary data interaction, thereby reducing network transmission overhead and delay. Considering the key role of indexes in query performance, index weights are introduced to guide the optimization of indexes to improve query speed. By identifying and processing key dependencies, and optimizing data flow and interaction patterns, the robustness of the system can be enhanced. Accurate performance evaluation results can reduce waiting time and improve query response speed.

[0130] In a specific embodiment of the present invention, a data lineage graph is constructed according to the query plan, which includes the dependencies and flow paths between data, collects and counts the data volume of each dependency (i.e., the node in the data lineage graph), the average size of the record, and the frequency of data interaction between the dependencies, and sets the weight coefficient according to actual needs; for each query plan, the number of data entries and the average size of each record are determined, and the contribution of this part to the performance evaluation is calculated using a formula to obtain the performance evaluation results of each query plan.

[0131] In a preferred embodiment of the present invention, according to the evaluation results, the data item dependencies and expressions of the query solution are obtained, and the submodules of the lineage information are extracted to construct the data lineage information, including:

[0132] According to the evaluation results, determine the initial input data item set and the output data item set;

[0133] Analyze the data flow information provided in the evaluation results and establish direct mapping relationships between data items;

[0134] Using the direct mapping relationship, starting from the output data item, reversely search for all input data items that affect its generation, and starting from the input data item, forward search for all output data items derived from it;

[0135] For each relevant data item found, a Cartesian product operation is performed to obtain a Cartesian product operation result;

[0136] Based on the results of the Cartesian product operation, complete data lineage information is generated, which depicts the dependencies and transformation paths of all data items from the original input to the final output.

[0137] In an embodiment of the present invention, by determining the input and output data item sets and analyzing the data flow information, a clear data flow view can be provided for the user, which helps the user understand the complete process of data from input to output, so as to better manage and control the data flow. By using the direct mapping relationship, the output data item can be reversely searched to the input data item, and the input data item can be forward searched to the output data item, so that the dependency relationship between the data items can be accurately analyzed. By performing Cartesian product operations, all possible combination relationships between data items can be revealed. When there is a problem with the data, the complete data lineage information can quickly locate the source of the problem, improve the efficiency of data problem troubleshooting and resolution, reduce business interruptions and losses caused by data errors, and help maintain data consistency and integrity. Data lineage information provides strong support for data governance. By understanding the dependencies and conversion paths between data, organizations can more effectively formulate and implement data standards, policies and processes, thereby improving the effectiveness of data governance. Complete data lineage information helps enterprises better understand their data assets and make more informed data-driven decisions.

[0138] In a specific embodiment of the present invention, the initial input data item set refers to the data items initially received or used in the query or data processing process; the output data item set refers to the data items that are finally generated or output after a series of processing. The evaluation results include information about query performance, data dependency, conversion efficiency, etc. According to the evaluation results, the system identifies and extracts all data items that serve as initial inputs to form an input data item set. Similarly, the system also identifies and extracts all final output data items to form an output data item set. The system carefully analyzes the data flow information in the evaluation results to identify how the data items directly affect each other. For example, a certain input data item directly determines the value of a certain output data item. These direct relationships are established and recorded; for each output data item, the system uses the established direct mapping relationship to reversely trace all input data items that have a direct impact on its generation, and then obtain which original data has a direct impact on the final result. For each input data item, the system also uses the direct mapping relationship to forwardly trace all output data items directly or indirectly derived from it to obtain the process of how the original data is propagated and transformed in the entire processing flow.

[0139] In mathematics and computer science, the Cartesian product of two sets is a set consisting of all possible ordered pairs in the two sets. In the above steps, the Cartesian product is used to represent all possible combinations between data items. For each related data item (whether input or output) searched through the above reverse and forward search, the system performs a Cartesian product operation to reveal all possible combinations and dependencies between data items, especially when multiple inputs or multiple outputs are involved. Based on the results of the Cartesian product operation, the system generates a comprehensive data lineage graph, which not only shows how each data item is related to other data items, but also details the complete transformation path of the data from input to output.

[0140] like Figure 2 As shown, a data processing method for water conservancy heterogeneous big data components, the method comprising:

[0141] Step 21, processing the data lineage of multiple big data components in the same process to obtain data processing tasks;

[0142] Step 22, converting the data processing task into data in SQL format, and preprocessing the data in SQL format to obtain preprocessed data;

[0143] Step 23, generating an abstract syntax tree from the preprocessed data, and converting the abstract syntax tree into a parse tree to generate a query solution;

[0144] Step 24, filtering the query solutions to obtain filtered query solutions, and evaluating the filtered query solutions to obtain evaluation results;

[0145] Step 25, based on the evaluation results, obtain the data item dependencies and expressions of the query solution, extract the sub-modules of the lineage information, and construct the data lineage information.

[0146] In an embodiment of the present invention, the same process is used to process data lineage for multiple big data components, ensuring the consistency and standardization of data processing. The data processing task is converted into data in SQL format so that data from different sources and structures can be processed under a unified query language framework. By generating an abstract syntax tree from the preprocessed data and converting it into a parse tree, a query solution can be generated more accurately. By filtering and evaluating the generated query solution, the query solution with the best performance and the most in line with the needs can be quickly screened out. According to the evaluation results, the data item dependencies and expressions of the query solution are obtained, and the sub-modules of the lineage information are extracted, so that comprehensive and detailed data lineage information can be constructed. Complete data lineage information improves the traceability and transparency of the data. When there is a problem with the data or an audit is required, the data source and flow path can be quickly located, improving the efficiency and accuracy of problem solving. Through detailed data lineage information, the flow of sensitive data can be more easily identified and monitored to ensure the security and compliance of the data.

[0147] It should be noted that the system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0148] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing system for water conservancy heterogeneous big data components, characterized by: The system comprises: The processing module is used to process the data lineage of various big data components in the same process to obtain data processing tasks; A structure acquisition module is used to convert the data processing task into data in SQL format and preprocess the data in SQL format to obtain preprocessed data; A structure conversion module, used to generate an abstract syntax tree from the preprocessed data, and convert the abstract syntax tree into a parse tree to generate a query solution; A filtering module, used for filtering query solutions to obtain filtered query solutions, and evaluating the filtered query solutions to obtain evaluation results; The lineage building module is used to obtain the data item dependencies and expressions of the query solution according to the evaluation results, and extract the sub-modules of the lineage information to build the data lineage information; The same process is used to process the data lineage of various big data components to obtain data processing tasks, including: Get the environmental status data of the big data component; Based on the environmental status data of the big data component, Calculated environmental status assessment index ,in, Indicates the number of valid data entries that have been verified; Indicates the total number of data entries in the dataset; represents the weight coefficient; Indicates The value of a key indicator; Indicates the number of indicators; Indicates the number of data entries that meet the criteria; , and represents the weight coefficient; Represents an index, used to traverse all key indicators; According to historical data, Calculate the threshold of the environmental status evaluation index when the big data component is operating normally ,According to the environmental status evaluation index and threshold, the normal environmental status of the big data component is determined, where, It represents the mean value of the environmental status assessment index in historical data; represents the standard deviation of the environmental status assessment index; Represents the total number of environmental status assessment indices in the historical data set; Represents an index, which is used to traverse each data point in the historical data set; Indicates The value of the historical environmental status assessment index; , and represents weight; According to the normal environmental status of the big data component, the data lineage is processed with the same process to obtain the data processing task.

2. The data processing system for water conservancy heterogeneous big data components according to claim 1 is characterized in that: Convert the data processing task into SQL format and preprocess the data in SQL format to obtain preprocessed data, including: Parse the data processing tasks and obtain the instruction set of the big data component; Read data and convert data processing tasks into SQL format data according to the instruction set of the big data component; Perform data cleansing on the data in SQL format to obtain cleaned data; The cleaned data is passed through Perform filtering to obtain filtered data, wherein: It's at the time The filtered data, is the cleaned data sequence at time point The data value of It is at every point in time; is the index; The filtered data is normalized to obtain preprocessed data.

3. The data processing system for water conservancy heterogeneous big data components according to claim 2 is characterized in that: Generate an abstract syntax tree from the preprocessed data and convert the abstract syntax tree into a parse tree to generate a query solution, including: Scan the preprocessed data using a scanner, and decompose the preprocessed data into a token sequence, where each token is an atomic element in the preprocessed data, including a keyword, an identifier, an operator, or a value; The token sequence is converted into an abstract syntax tree through a recursive descent parser. In the abstract syntax tree, each node represents an instance of a grammatical structure, and the node type reflects the type of the grammatical structure. Take the abstract syntax tree as an individual in the initial population and define a fitness function for evaluating the quality of each abstract syntax tree individual corresponding to the query solution; According to the fitness function, the corresponding abstract syntax tree individual is selected as the parent, two parent abstract syntax tree individuals are randomly selected, and a new abstract syntax tree individual is generated by exchanging subtrees or nodes; the selected abstract syntax tree individual is randomly modified, and the child abstract syntax tree individual generated by the genetic operation is converted into a query plan, and the query plan is continuously optimized through multiple rounds of genetic iterations until a preset number of iterations is reached to obtain the final abstract syntax tree individual; Convert the final abstract syntax tree into a specific query plan.

4. The data processing system for water conservancy heterogeneous big data components according to claim 3 is characterized in that: The calculation formula of the fitness function is: ； in, The weight representing the execution efficiency; Indicates the number of records returned by error; represents the weight of accuracy; , and represents weight; , , , , , , and They represent the maximum query execution time, actual execution time, number of correct records, total number of records, number of error records, number of join operations, number of subqueries, and number of nested levels.

5. The data processing system for water conservancy heterogeneous big data components according to claim 4 is characterized in that: Convert a sequence of tokens into an abstract syntax tree via a recursive descent parser consisting of: Starting from the beginning of the token sequence, a recursive descent parser is used for parsing; the parser reads and matches tokens one by one according to the grammar rules, and gradually builds each node of the abstract syntax tree; When a complete grammatical structure is matched, a new abstract syntax tree node is created and its type is set; For each newly created node, if it contains substructures, recursively call the parser to process the substructures and add the substructures as child nodes to the current node. This process continues until the token sequence is completely consumed or an unmatched token is encountered. After the abstract syntax tree is constructed, a post-processing phase is started, starting from the root node of the abstract syntax tree, and performing a depth-first traversal. During the traversal, for each accessed node, a metadata query operation is performed according to its type and content. The query operation involves searching the symbol table and accessing the database. If valid metadata is found, assign it to the metadata field of the current abstract syntax tree node. If the abstract syntax tree node does not require metadata or the metadata query fails, keep the empty value or set the default value. When the entire abstract syntax tree is traversed and all required metadata are queried and assigned, an abstract syntax tree containing complete structure and metadata information is output.

6. The data processing system for water conservancy heterogeneous big data components according to claim 5 is characterized in that: The query scheme is filtered to obtain a filtered query scheme, and the filtered query scheme is evaluated to obtain an evaluation result, including: Read all pending query plans and store them in a processing queue; Call the syntax parser to perform syntax checks on the query plans in the processing queue one by one, mark and remove those query plans that do not meet the syntax specifications, so that the remaining query plans are grammatically correct; Perform logical verification on query plans that pass the syntax check, identify query plans that are logically unreasonable or have potential errors, and remove query plans with logical problems from the processing queue; Hash the remaining query solutions and generate a unique hash value for each query solution; Compare hash values ​​to find and remove duplicate query solutions so that each query solution is unique in the processing queue; Automatically parse the table names, field names and associated relationships in the remaining query solutions; build an internal data model based on the table names, field names and associated relationships in the remaining query solutions; Analyze the internal data model to obtain the dependencies and flow paths between data; Based on the dependencies and flow paths between data, a data lineage graph is constructed to show the association and flow between data; Based on the data lineage graph and data volume, the performance of each query solution is evaluated to obtain the evaluation results.

7. The data processing system for water conservancy heterogeneous big data components according to claim 6 is characterized in that: Based on the data lineage graph and data volume, each query solution is evaluated to obtain the evaluation results, including: Based on the data lineage graph and data volume, each query solution is Conduct performance evaluation to obtain evaluation results; in, represents the performance evaluation value; , , and represents the weight coefficient; Indicates the number of data entries; Indicates the average size of each record; Indicates the number of dependencies; Indicates The data volume weight of each dependency; Indicates The amount of data associated with each dependency; Indicates the total number of dependencies; Indicates The coupling weight of each dependency; Indicates Dependencies related to The connection strength of each dependency; Indicates The processing time weight of each dependency; Indicates processing The time required for each dependency; ,in, is the number of direct connections, indicating dependencies and dependencies The number of direct relationships between is the frequency of data interaction, indicating the dependency and dependencies The frequency of data exchange between them; Representing dependencies The weight of Represents an index.

8. The data processing system for water conservancy heterogeneous big data components according to claim 7 is characterized in that: According to the evaluation results, the data item dependencies and expressions of the query solution are obtained, and the sub-modules of the lineage information are extracted to construct the data lineage information, including: According to the evaluation results, determine the initial input data item set and the output data item set; Analyze the data flow information provided in the evaluation results and establish direct mapping relationships between data items; Using the direct mapping relationship, starting from the output data item, reversely search for all input data items that affect its generation, and starting from the input data item, forward search for all output data items derived from it; For each relevant data item found, a Cartesian product operation is performed to obtain a Cartesian product operation result; Based on the results of the Cartesian product operation, complete data lineage information is generated, which depicts the dependencies and transformation paths of all data items from the original input to the final output.

9. A data processing method for water conservancy heterogeneous big data components, characterized in that: Applied to the system according to any one of claims 1 to 8, the method comprises: Process the data lineage of multiple big data components in the same process to obtain data processing tasks; Convert the data processing task into data in SQL format, and preprocess the data in SQL format to obtain preprocessed data; Generate an abstract syntax tree from the preprocessed data, and convert the abstract syntax tree into a parse tree to generate a query solution; Filtering the query solutions to obtain filtered query solutions, and evaluating the filtered query solutions to obtain evaluation results; According to the evaluation results, the data item dependencies and expressions of the query solution are obtained, and the sub-modules of the lineage information are extracted to construct the data lineage information.

Images