Data processing method and system based on business intelligent system

Abstract

The invention provides a data processing method and system based on a business intelligence system. The method comprises the following steps: receiving request data, and standardizing the request data to obtain standardized data; analyzing the standardized data based on a pre-constructed dynamic semantic map to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities and business concepts; performing logic compiling on the analysis result to generate an executable operation chain; the executable operation chain comprises sequenced atomic operation steps, and the atomic operation steps comprise data processing logic and metadata binding information; and running the executable operation chain to obtain an analysis result corresponding to the request data. The method comprises the following steps: analyzing request data based on a dynamic semantic map; and the analysis result is logically compiled, an executable operation chain is generated, and the data processing process is decomposed into sequenced atomic operation steps, so that the data processing flow becomes clear, and the accuracy and efficiency of request data processing are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a data processing method and system based on a business intelligent system. BACKGROUND

[0002] Business Intelligence (BI) is a technology that transforms existing data in an enterprise into knowledge to help the enterprise make wise business decisions. It includes various software tools, methods and processes. For example, through technologies such as data warehouse, data mining, online analytical processing (OLAP), data from multiple data sources (such as the enterprise's internal sales system, financial system, production system, etc.) is collected, stored, managed and analyzed.

[0003] With the explosive growth of enterprise data analysis needs and the popularity of natural language interaction technology, when the current enterprise-level BI system responds to business needs, the analysis needs proposed by business personnel using natural language cannot be accurately recognized by general natural language processing models due to the inclusion of private terms and industry-specific expressions, and the existing technology lacks the ability to automatically compile business needs into executable data operation chains, still requiring manual construction, which is inefficient and consumes a large amount of manpower. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a data processing method and system based on a business intelligent system to improve the efficiency and accuracy of demand analysis.

[0005] In a first aspect, the embodiments of the present application provide a data processing method based on a business intelligent system, comprising: receiving request data, standardizing the request data to obtain standardized data; analyzing the standardized data based on a pre-constructed dynamic semantic graph to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities and business concepts; logically compiling the analysis result to generate an executable operation chain; the executable operation chain includes sequential atomic operation steps, and the atomic operation steps contain data processing logic and metadata binding information; running the executable operation chain to obtain an analysis result corresponding to the request data.

[0006] The embodiment of the application can standardize different sources, different formats and different expressions of data, so as to facilitate subsequent analysis. Based on dynamic semantic graph analysis, various business terms and data entities in the standardized data can be accurately and quickly understood and disassembled. Then, the analysis result is logically compiled to generate an executable operation chain, so that the data processing process is decomposed into sequential atomic operation steps, the data processing flow becomes clear, and the accuracy and efficiency of the request data processing are improved.

[0007] In a possible implementation of the first aspect, the standardized data is analyzed based on the pre-constructed dynamic semantic graph to obtain an analysis result, including: matching the standardized data with the dynamic semantic graph, if the matching is successful, extracting a target semantic structure matched successfully from the dynamic semantic graph; generating a structured analysis tree based on the target semantic structure; performing intent classification and entity link analysis based on the structured analysis tree to obtain an intent category and a standardized entity; performing semantic relation construction according to the intent category and the standardized entity to obtain the analysis result.

[0008] The embodiment of the application can accurately identify the semantic information contained in the data by matching the standardized data with the dynamic semantic graph, so that the subsequent analysis work can be based on accurate semantic basis, reducing the risk of semantic understanding deviation and error, and improving the accuracy of data analysis.

[0009] In a possible implementation of the first aspect, the analysis result is logically compiled to generate an executable operation chain, including: extracting semantic elements from the analysis result, converting the semantic elements into a relational algebra expression, and generating an abstract syntax tree based on the relational algebra expression; converting the abstract syntax tree into a logical plan expression; generating an optimization space according to the logical plan expression, and generating a cost function according to the abstract syntax tree; performing dynamic programming according to the optimization space and the cost function to obtain an optimal plan; mapping the logical operation method in the optimal plan into a physical operation type, and generating a directed acyclic graph based on the data dependency relationship in the optimal plan; converting the physical operation type into executable code; and generating an executable operation chain based on the executable code and the directed acyclic graph.

[0010] The embodiment of the application realizes automatic generation of an executable operation chain by converting semantic elements in a parsing result into a relational algebra expression and generating an abstract syntax tree, and uses an optimization space and a cost function to perform dynamic planning, thereby improving the efficiency of processing request data.

[0011] In a possible implementation of the first aspect, the executable operation chain is run to obtain an analysis result corresponding to the request data, including: An execution engine is allocated to an atomic operation step in the executable operation chain through intelligent routing; The corresponding principle operation step is processed by the execution engine to obtain a processing result; The processing result of the execution engine is fused to obtain the analysis result.

[0012] The embodiment of the application allocates an atomic operation step to a corresponding execution engine based on intelligent routing, for example, allocating to an engine with high computing power for a computationally complex request, allocating to a cache service for a simple query, and allocating to a stream processing engine for a request with high real-time requirement. After the engines are processed, the results are uniformly encapsulated to obtain a final result. Therefore, the intelligent routing method allocates corresponding engines according to actual requirements, thereby improving the efficiency and accuracy of data processing.

[0013] In a possible implementation of the first aspect, the request data is standardized, including: The data type of the request data is identified, and the request data is converted into text stream data based on the data type; A business term to be standardized in the text stream data is extracted, and the business term to be standardized is matched with a term in a cache. If the matching is successful, the business term to be standardized is replaced with a standard term corresponding to the term in the cache; If the matching is unsuccessful, the business term to be standardized is matched with a pre-constructed business term graph. If the matching is successful, a standard term is obtained, and the business term to be standardized is replaced with the standard term; If the matching is unsuccessful, the business term to be standardized is subjected to fuzzy matching.

[0014] The embodiment of the application converts the request data into text stream data, and then matches the terms in the cache. If the matching is unsuccessful, the business term graph is matched. If the matching is still unsuccessful, the business term is subjected to fuzzy matching. In the matching process, the matching efficiency and accuracy are taken into account.

[0015] In a possible implementation of the first aspect, the method further includes: If a new business term is detected, a subgraph related to the new business term is extracted from a dynamic semantic graph; The subgraph-based topology feature calculates a relationship probability between the new business term and the business term in the subgraph; A candidate triple is generated according to the relationship probability; the candidate triple includes the new business term, the business term in the subgraph, and the association relationship; A confidence degree of the new business term is calculated based on the candidate triple; if the confidence degree is greater than a preset threshold, the new business term is added to the dynamic semantic graph based on the candidate triple.

[0016] Embodiments of the present application detect a new term, update the dynamic semantic graph after detecting the new term, and facilitate subsequent accurate semantic analysis.

[0017] In a possible implementation manner of the first aspect, the method further includes: Each department calculates a model gradient based on local data; The model gradient is encrypted, and the encrypted model gradient is uploaded to a secure aggregation node; The encrypted model gradient of each department is fused by the secure aggregation node to obtain fused data; The semantic understanding model is updated based on the fused data; wherein the semantic understanding model is used to analyze the standardized data based on the pre-constructed dynamic semantic graph to obtain an analysis result.

[0018] Embodiments of the present application fuse the encrypted model gradient of each department by the secure aggregation node, and update the semantic understanding model based on the fused data, thereby improving the accuracy of subsequent semantic understanding model analysis.

[0019] In a possible implementation manner of the first aspect, the method further includes: After listening to the metadata change event of the business system, the affected model is determined based on the metadata change event and the pre-constructed blood relation dependency graph; If the semantic understanding module is the affected model, the semantic understanding module is retrained; If the logic compilation module is the affected model, the executable operation chain is reconstructed.

[0020] Embodiments of the present application automatically trigger the reconstruction and verification of the business intelligent system after listening to the metadata change event of the business system, so that the analysis logic and the business rules are real-time synchronized.

[0021] In the second aspect, embodiments of the present application provide a business intelligent system, including: A semantic standardization module is configured to receive request data, standardize the request data, and obtain standardized data; The semantic understanding module is configured to analyze the standardized data based on a pre-constructed dynamic semantic graph to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities, and business concepts; The logic compiling module is configured to compile the analysis result to generate an executable operation chain; the executable operation chain includes sequential atomic operation steps, and the atomic operation steps include data processing logic and metadata binding information. The physical execution module is configured to run the executable operation chain to obtain an analysis result corresponding to the request data.

[0022] In a third aspect, an electronic device is provided, including: a processor, a memory, and a bus, wherein: The processor and the memory complete mutual communication through the bus; The memory stores program instructions that can be executed by the processor, and the processor calling the program instructions can execute the method of the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium, including: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method in any possible implementation manner of the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer program instructions, and the computer program instructions are read and run by the processor to execute the method in any possible implementation manner of the first aspect.

[0025] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be understood through implementation of the embodiments of the present application. The purpose and other advantages of the present application can be achieved and obtained through the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0027] Figure 1 A data processing method flow diagram based on a business intelligent system is provided for the embodiments of the present application; Figure 2 A semantic analysis method flow diagram is provided for the embodiments of the present application; Figure 3 A flowchart of a method for generating an executable operation chain is provided for an embodiment of the present application. Figure 4 A flowchart of a method for running an executable operation chain is provided for an embodiment of the present application. Figure 5 An abnormality corresponding measure diagram is provided for an embodiment of the present application. Figure 6 A flowchart of a method for standardizing terms is provided for an embodiment of the present application. Figure 7 A flowchart of a method for resolving data conflicts is provided for an embodiment of the present application. Figure 8 A flowchart of a method for updating a dynamic voice graph is provided for an embodiment of the present application. Figure 9 A flowchart of a method for optimizing a semantic understanding model is provided for an embodiment of the present application. Figure 10 A flowchart of a method for updating a system is provided for an embodiment of the present application. Figure 11 A flowchart of a structure of a business intelligent system is provided for an embodiment of the present application. Figure 12 An electronic device entity structure diagram is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments of the present application, and are not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion.

[0030] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0031] Reference to“an embodiment” herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment. The appearances of the phrase“in an embodiment” in various places in the specification are not necessarily referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. As those skilled in the art will appreciate, the embodiments described herein can be adapted to a wide variety of alternative embodiments.

[0032] In the description of the embodiments of the present application, the term“and / or” only means an association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character“ / ” herein generally means that the front and rear associated objects have an“or” relationship.

[0033] In the description of the embodiments of the present application, the term“a plurality of” means more than two (including two), and similarly, “a plurality of groups” means more than two groups (including two groups), and “a plurality of pieces” means more than two pieces (including two pieces).

[0034] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms“mounting”,“connection”,“connection”,“fixing” and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0035] The current business intelligence system generally relies on the following three types of known technologies to realize semantic understanding and modeling: 1) Static semantic dictionary technology Through the pre-defined mapping table of business terms and technical fields (such as“sales” mapped to the sales_amount field), basic natural language queries are supported. This technology requires manual maintenance of the glossary and cannot adapt to new terms added by the enterprise.

[0036] 2) Visual drag-and-drop modeling technology Allows users to associate data tables and define calculation indicators through a graphical interface (such as connecting the product table and the sales table by dragging and dropping, and setting the profit rate = (sales - cost) / sales). Although it reduces the threshold of SQL programming, it still requires manual completion of logical design.

[0037] 3) General NLP model integration technology An open-source pre-trained language model is used to parse entities and intents in user queries and generate a preliminary SQL skeleton. Due to the lack of domain knowledge adaptation, the accuracy of professional term recognition is limited.

[0038] Therefore, the existing business intelligence system has the problems of inaccurate semantic analysis, inability to automatically derive the data correlation path and calculation logic of complex business requirements, and the need for manual parameters, so the efficiency and accuracy of requesting data processing are low.

[0039] To solve the above technical problems, the embodiments of the present application provide a data processing method based on a business intelligence system. The method realizes data analysis through a dynamic semantic graph, automatically generates an executable operation chain based on the analysis result, and then runs the executable operation chain to obtain a processing result, thereby improving the processing efficiency and accuracy of requesting data.

[0040] It can be understood that the data processing method based on the business intelligence system provided by the embodiments of the present application can be applied to an electronic device, which includes a terminal and a server. The terminal can be a smart phone, a tablet computer, a computer, a personal digital assistant (PDA), etc. The server can be an application server or a Web server. The business intelligence system is running in the electronic device. For ease of description, the business intelligence system is described as the execution subject in the following.

[0041] Figure 1 A data processing method based on a business intelligence system provided by the embodiments of the present application is shown in a flowchart as shown in Figure 1 The method comprises the following steps: Step 101: receiving request data, standardizing the request data to obtain standardized data; Step 102: analyzing the standardized data based on a pre-constructed dynamic semantic graph to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities and business concepts; Step 103: logically compiling the analysis result to generate an executable operation chain; the executable operation chain includes sequential atomic operation steps, and the atomic operation steps contain data processing logic and metadata binding information; Step 104: running the executable operation chain to obtain an analysis result corresponding to the request data.

[0042] In step 101, the request data can be input by the user through a web interface, a mobile application, or an API interface. The form of the input request data can include text, voice, and files, etc. In addition to the variety of forms, the content contained in the request data is also diverse, for example: there may be special characters, slang, dialect, sensitive words, etc. in the request data. In order to improve the efficiency and accuracy of subsequent processing, the received request data can be standardized.

[0043] After the business intelligence system (referred to as the system) receives the request data, the following standardization processing can be performed: Data format unification: convert data of different formats into a unified internal data format of the system. For example, convert data in JSON, CSV, XML formats into table format in a relational database, or into a specific serialization format (such as Parquet, ORC) for easy storage and query.

[0044] Data encoding conversion: encode and convert text information in the data to ensure that all text data uses a unified character encoding (such as UTF-8). This can avoid data garbled problems caused by inconsistent encoding, especially when processing information in different languages and regions.

[0045] Data structure standardization: for structured data, ensure that it conforms to the predefined schema (schema), including field type, length, constraint condition, etc. For example, ensure that the date field is in the format "YYYY-MM-DD", and the amount field is of numeric type and retains two decimal places. For unstructured data (such as text, image), preliminary classification and labeling may be needed for subsequent processing.

[0046] Data cleaning: remove noise, redundancy, and error information in the data. For example, identify and process missing values (such as filling with average value, median, or deleting missing records), correct errors (such as correcting spelling errors, data inconsistencies, etc.), to improve data quality.

[0047] Data security and compliance check: check the security and compliance of the data to ensure that the data complies with the enterprise's security policy and relevant laws and regulations (such as data protection regulations). This includes encrypting sensitive data, desensitizing processing, and preliminary control of data access permissions.

[0048] After the above processing, the input content is encapsulated into a unified message structure, including the original text, session ID (used to associate the context), and metadata, i.e. standardized data.

[0049] In step 102, the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities, and business concepts. This dynamic semantic graph can provide a rich semantic knowledge base for the business intelligence system.

[0050] Business terms include various professional terms, business process names, product names, etc. used within the enterprise. These terms are frequently used by business personnel in their daily work and can accurately describe the business activities and data of the enterprise. For example, in the financial industry, business terms may include "loan approval", "credit scoring", "risk management", etc.; in the network sales industry, there may be "commodity promotion", "inventory turnover rate", "customer loyalty", etc.

[0051] Data entities refer to specific objects or concepts in enterprise data, usually corresponding to business terms. For example, "customer" is a data entity, and its attributes may include customer ID, name, contact information, purchase history, etc.; "order" is also a data entity, containing order number, order date, customer ID, product details, amount, etc. The relationship between data entities is also an important part of the dynamic term graph, such as "customer" under "order", "order" containing "product", etc.

[0052] Business concepts are higher-level abstractions and generalizations in the business domain, reflecting the nature and laws of business. For example, "supply chain management" is a business concept that covers procurement, production, logistics, inventory, and other business processes and data entities. "Customer relationship management" is also a business concept that involves customer acquisition, customer maintenance, customer service, and other aspects. Business concepts can help the system understand and process data from a more macro perspective.

[0053] The dynamic term graph organizes these business terms, data entities, and business concepts in the form of a directed attribute graph. The directed edges represent the semantic relationships between them, such as "inheritance", "association", "belongs to", etc. For example, "customer" belongs to the business concept of "business object"; there is an association between "order" and "customer" for "ordering". At the same time, each node (representing a term, entity, or concept) and edge can have attributes to describe its characteristics and properties. For example, the "customer" node can have attributes such as "type" (such as individual customer, enterprise customer), "value level", etc.; the "order" edge can have attributes such as "order date", "order amount", etc. The dynamic term graph can be dynamically updated as the enterprise business develops and changes to reflect new business terms, entities, and concepts, as well as changes in their relationships.

[0054] The standardized data is matched with the dynamic term graph. The system searches for semantic elements in the graph that match the keywords, patterns, structures, and other features in the data. For example, if the standardized data contains information such as "customer purchases product A", the system searches for terms, entities, and concept nodes related to "customer", "purchase", and "product A" in the dynamic term graph and determines whether their relationships are consistent with the existing semantic relationships in the graph. When a match is found, the system extracts the target semantic structure from the dynamic term graph that corresponds to the matching data. This target semantic structure contains business terms, data entities, business concepts, and their semantic associations related to the data, accurately abstracting and expressing the semantics of the data. If no existing terms are matched, a fuzzy reasoning mode is started, and the target semantic structure is obtained by analyzing the context semantics through a neural network.

[0055] Then, a structured parse tree is generated according to the target semantic structure, followed by intent classification (determining analysis types such as statistics, comparison, etc.) and entity linking (mapping natural language expressions to specific data entities). After completing semantic relationship construction (establishing logical associations between indicators, dimensions, and filtering conditions) according to the intent type and entities, the parse result is output. It should be noted that the parse result includes the logical associations between indicators, dimensions, and filtering conditions.

[0056] In step 103, after obtaining the parse result, the parse result can be logically compiled. Specifically, first, the semantic elements are converted to an intermediate representation form through a relational algebra converter, then the optimizer reorganizes and optimizes the operation order based on a cost model, and finally an executable operation chain is generated according to the specificity of the target execution engine. The operation chain consists of sequential atomic operation steps, each containing complete data processing logic and metadata binding information, and the final output is a standardized operation chain. The executable operation chain specifies the specific execution flow and operation order of data processing, ensuring that the system can efficiently execute data processing tasks according to the optimized plan.

[0057] In step 104, each atomic operation step is executed in sequence according to the order and logic specified in the executable operation chain. The executable code of each step runs in the execution engine and processes the data accordingly. During execution, the system ensures that data flows and transfers correctly between operations according to the data dependency relationships in the executable operation chain. For example, the data reading operation is performed first to load data into memory, then the data is filtered according to the filtering conditions, then the filtered data is connected with other related data, and finally the aggregation calculation is performed to obtain the final analysis result.

[0058] After executing all atomic operation steps, the system generates analysis results corresponding to the requested data. These results can be presented in various formats, such as tables, charts, and reports, depending on business needs and user settings. These results reflect in-depth analysis and insights into enterprise data, providing strong support for decision-making and business optimization. For example, generating a sales data analysis report can showcase sales trends for different products, customer purchasing behavior analysis, and regional sales differences, helping businesses understand market dynamics and develop marketing strategies and product plans.

[0059] This application's embodiments standardize request data, enabling data from different sources, formats, and expressions to be unified and standardized, facilitating subsequent parsing. Based on dynamic semantic graph parsing, it can accurately and quickly understand and decompose various business terms and data entities in the standardized data. Then, the parsing results are logically compiled to generate an executable operation chain, decomposing the data processing process into sequential atomic operation steps, making the data processing flow clear and improving the accuracy and efficiency of request data processing.

[0060] Based on the above embodiments, the standardized data is parsed using a pre-constructed dynamic semantic graph to obtain the parsing results, including: The standardized data is matched with the dynamic semantic graph. If the match is successful, the target semantic structure that was successfully matched is extracted from the dynamic semantic graph. Generate a structured parse tree based on the target semantic structure; Intent classification and entity link analysis are performed based on structured parse trees to obtain intent categories and standardized entities; Semantic relationships are constructed based on intent categories and standardized entities to obtain parsing results.

[0061] In the specific implementation process Figure 2 This is a schematic diagram of a semantic parsing method provided in an embodiment of this application, such as... Figure 2 As shown. After obtaining standardized data, the standardized data is matched with a dynamic terminology graph. The system searches for matching semantic elements in the dynamic semantic graph based on features such as keywords, patterns, and structures in the data. For example, if the standardized data contains information such as "customers purchase product A," the system searches the dynamic terminology graph for terms, entities, and concept nodes related to "customers," "purchases," and "product A," and determines whether the relationships between them are consistent with the existing semantic relationships in the graph.

[0062] When a match is successful, the system extracts the target semantic structure corresponding to the matched data from the dynamic terminology graph. This target semantic structure contains business terms, data entities, business concepts related to the data, and their semantic associations, which is an accurate abstraction and expression of the data semantics.

[0063] Based on the extracted target semantic structure, the system generates a structured parse tree. The structured parse tree is a tree-like data structure that represents the semantic hierarchy and relationships of the data in the form of a tree. The root node of the tree can represent the semantic theme or business intent of the entire data, the intermediate nodes represent the main semantic components in the data and their relationships, and the leaf nodes correspond to specific entities, attributes, or concepts in the data. For example, for the data "customer purchases product A", the root node of the structured parse tree may be "purchase behavior", its child nodes include "customer" (containing customer ID, name, etc. attributes) and "product A" (containing product ID, name, price, etc. attributes), and their "purchase" relationship (containing purchase quantity, purchase date, etc. attributes). This structured representation helps to more clearly show the semantic hierarchy and logical relationships of the data.

[0064] The system further analyzes the generated structured parse tree, including intent classification and entity linking analysis. Intent classification refers to determining the business intent or purpose expressed by the data, such as querying certain information, performing data analysis, generating reports, or executing certain business operations, etc. In identifying the analysis type of the user input request data, a BERT+GAT dual feature fusion architecture can be used, that is, BERT is used to extract semantic features for the structured parse tree , while the dynamic terminology graph encodes the structural features through the graph attention network ; after combining them in the feature fusion layer, the intent category is classified and output through the fully connected layer. The embodiment of the present application realizes the end-to-end conversion of natural language to complete operation chain through a two-stage compilation engine, and the logical completeness is greatly improved.

[0065] Entity linking analysis is to associate and link the entities in the data with the standard entities in the dynamic terminology graph, ensuring that the system can accurately understand the entity objects involved in the data. For example, in the data "customer purchases product A", through intent classification, it can be determined that its intent is to record a sales transaction; through entity linking analysis, "customer" is linked to the standard information of the customer in the graph (such as customer profile), and "product A" is linked to the detailed information of the product in the graph (such as product specifications, inventory information, etc.).

[0066] The specific implementation path of entity linking is as follows: 1. Candidate generation: quickly retrieve possible matching graph entities based on term index; Build an inverted index wherein key is a surface form value is a set of entities satisfies: .

[0067] 2. Disambiguation ranking: combine contextual similarity, entity popularity, etc. to rank candidates Given mention and its context , the ranking score of candidate entity : wherein denotes a BERT-based contextual encoding similarity function denotes historical click rate or business predefined priority.

[0068] 3. Value domain verification: filter illegal mappings according to domain rules.

[0069] def validate_value(table, column, value): sql = f"SELECT 1 FROM {table} WHERE {column}=? LIMIT 1" return db.execute(sql, [value]).fetchone() is not None The final output uniquely determines the standardized entity, ensuring the accuracy of downstream analysis.

[0070] According to the results of intent classification and entity linking analysis, the system constructs the semantic relationship between each element in the data, thereby obtaining the analysis result. This step aims to more comprehensively understand and express the business meaning of the data, and organically combines each part of the data to form a complete semantic network. For example, in the above sales transaction data, the purchase relationship between "customer" and "product A" is constructed, as well as the relationship between the purchase behavior and the transaction time, amount, etc. information, thereby forming a complete semantic description of the sales business.

[0071] The embodiments of the present application can accurately identify the semantic information contained in the data by matching standardized data with dynamic semantic graph, so that the subsequent analysis work can be based on accurate semantic basis, reducing the risk of deviation and error in semantic understanding, and improving the accuracy of data analysis.

[0072] On the basis of the above embodiments, the analysis result is logically compiled to generate an executable operation chain, including: extracting semantic elements from the parsing result, converting the semantic elements into a relational algebra expression, and generating an abstract syntax tree based on the relational algebra expression; converting the abstract syntax tree into a logical plan expression; generating an optimization space according to the logical plan expression and a cost function according to the abstract syntax tree; performing dynamic programming according to the optimization space and the cost function to obtain an optimal plan; mapping the logical operation in the optimal plan to a physical operation type and generating a directed acyclic graph based on the data dependency in the optimal plan; converting the physical operation type into executable code; and generating an executable operation chain based on the executable code and the directed acyclic graph.

[0073] In the specific implementation process, Figure 3 An executable operation chain generation method flowchart is provided for the embodiments of the present application, as Figure 3 shown. The detailed and specific implementation core technology path is as follows: 1) Intermediate representation generation The semantic parsing result is converted into an optimizable intermediate representation using a three-stage conversion method: Stage 1: Semantic element extraction The input semantic parsing result can be represented as:

[0074] That is, the parsing result includes a set of indicators, a set of dimensions, a set of filtering conditions, and a basic data relationship, and can also include the results of structured parsing tree, intent classification, entity link analysis, and semantic relationship construction. The semantic elements extracted from the parsing result are the core logical units of data processing, covering key information such as entities, attributes, business intent, and semantic relationships involved in the data. For example, in the parsing result of sales transaction data, the extracted semantic elements may include "customer" (entity), "product A" (entity), "purchase" (business intent and semantic relationship), "purchase quantity" (attribute), "purchase amount" (attribute), etc.

[0075] Stage 2. Relationship algebra conversion Relational algebra is a logical expression method based on sets, used to describe queries and operations on data in relational databases. By converting the semantic elements into relational algebra, business semantics can be converted into data operation logic. For example, for the above sales transaction data, converting to a relational algebra expression may involve selecting records that meet certain purchase conditions from the "customer" table and "product" table, and performing join operations to obtain related customer and product information, while calculating purchase quantity and amount, etc.

[0076] In the embodiments of the present application, the semantic elements can be converted into the following relational algebra expression:

[0077] wherein, represents a projection operation, retaining dimensions D and index M; represents a selection operation, applying filter condition F.

[0078] Stage 3. Abstract Syntax Tree Construction Based on the relational algebra expression, an abstract syntax tree (AST), i.e., an intermediate representation, is generated. The abstract syntax tree is a tree structure used to represent the syntax structure of code or logical expressions. In the embodiments of the present application, it represents the structured form of data processing logic. The generation of the abstract syntax tree facilitates the subsequent logical plan generation and optimization process. For example, the selection, projection, join, etc. operations in the relational algebra expression will become nodes in the abstract syntax tree, and the objects of the operations (such as table names, attribute names) and conditions will become child elements of the nodes. The generated optimizable tree structure can be represented as follows:

[0079] wherein, represents a projection node; represents a filter node; represents a data scan node.

[0080] 2) Execution Plan Optimization 1. From the abstract syntax tree (AST) to the optimizable logical plan, the logical plan is a plan representation closer to physical execution, which describes the logical steps and operation order of data processing. The logical plan expression is usually represented in the form of an operator tree, in which the operators (such as scan, select, join, aggregate, etc.) represent the basic operations of data processing, and the operators are connected through data flow. For example, the selection operation in the abstract syntax tree is converted into the "select" operator in the logical plan, which is connected with the corresponding table scan operator to form the logical flow of data processing.

[0081] Specifically, the AST can be converted into the following logical plan algebra expression:

[0082] 2. Cost-based optimization process According to the logical plan expression, an optimization space and a cost function are generated. The optimization space refers to the various different execution strategies and resource allocation schemes that can exist in the logical plan execution process. Therefore, the optimization space is the set of all execution plans equivalent to the original logical plan The cost function is used to evaluate the resource consumption (such as time, memory, I / O, etc.) and performance of different execution schemes. The cost function directly depends on the topology of the AST: wherein, represents the cost of computing node n, represents the statistical information extracted from the AST. By analyzing the operators in the logical plan and their attributes (such as data volume, table size, index situation, etc.), the system can determine possible optimization points and optimization spaces, and establish the corresponding cost function. For example, for a logical plan containing a join operation, the optimization space may include selecting different join algorithms (such as nested loop join, hash join, sort-merge join, etc.) or different join orders, and the cost function calculates the execution cost of each scheme according to data volume, memory availability, etc.

[0083] Finally, using the optimization space and cost function, a dynamic programming algorithm is used to solve the optimization, obtaining the optimal execution plan. That is, the solution formula is: .

[0084] 3) Executable operation chain generation Executable operation chain generation is the process of converting the optimized optimal plan into an executable physical operation sequence, and its core is to achieve cross-platform execution capability through hierarchical mapping and dynamic code generation.

[0085] The specific steps are as follows: First, according to the mapping rules specified by the system, the logical operators are mapped to the corresponding physical operation types, and based on the data dependency relationship, a directed acyclic graph (DAG) is generated by topological sorting. The directed acyclic graph is a graph structure used to represent task dependency relationship and execution order, in the data processing process, it represents the data flow direction and execution order between different physical operations. For example, in an execution plan containing multiple physical operations (such as data reading, filtering, joining, aggregation, etc.), the nodes in the directed acyclic graph represent physical operations, and the directed edges represent the data flow from the output of one operation to the input of another operation, while ensuring that there is no circular dependency in the entire graph, ensuring that the operations can be executed correctly in sequence. Subsequently, according to the characteristics of the target execution engine of the system and the optimization rules, the physical operations are adapted to the specific execution code of the engine, and finally a standardized operation chain composed of sequential atomic operations is generated, each operation node contains input and output metadata binding, ensuring that the original logical plan can strictly maintain semantic consistency during execution.

[0086] The embodiments of the present application convert the semantic elements in the parsing result into relational algebra expressions and generate an abstract syntax tree, use dynamic programming with optimization space and cost function, and realize the automatic generation of executable operation chain, improving the efficiency of processing the requested data.

[0087] On the basis of the above-mentioned embodiments, the executable operation chain is run to obtain an analysis result corresponding to the request data, including: An execution engine is assigned to an atomic operation step in the executable operation chain through intelligent routing; The corresponding principle operation step is processed through the execution engine to obtain a processing result; The processing result of the execution engine is fused to obtain the analysis result.

[0088] In the specific implementation process, Figure 4 The method flow diagram for running the executable operation chain provided by the embodiments of the present application is shown in Figure 4 After the executable operation chain is generated, the distributed scheduler can perform resource allocation and task decomposition according to the data size, calculation complexity, and time delay requirement, and then select the optimal execution engine through the intelligent routing mechanism. For simple queries, the cache service is directly called for fast response, for large data volume scenarios, automatic allocation is performed to the Spark cluster for batch processing, and for low delay requirements, the Flink stream processing engine is routed. After the execution engine completes the calculation, the processed data is visualized, the abnormal value is marked, and the summary is generated, and finally the analysis result is returned to the user end.

[0089] In another embodiment, explicit feedback and implicit behavior data of the user can also be continuously collected, where the implicit behavior data can be, for example, the user's click, the time spent on a certain page, etc. After the feedback information is encrypted and processed, it is returned to the knowledge graph and model training system of the semantic understanding module, forming a "execution-feedback-optimization" closed-loop learning mechanism, which greatly improves the average response speed of the system.

[0090] The embodiments of the present application assign the atomic operation steps to the corresponding execution engines based on intelligent routing, for example: for complex calculation requests, assign to an engine with high calculation capability; for simple queries, assign to a cache service; for high real-time requirements, assign to a stream processing engine. After each engine is processed, the results are uniformly packaged to obtain the final result. Therefore, this intelligent routing method assigns the corresponding engine according to the actual demand, improving the efficiency and accuracy of data processing.

[0091] On the basis of the above-mentioned embodiments, the request data is standardized, including: Identify the data type of the request data, and convert the request data to text stream data based on the data type; Extract the business term to be standardized in the text stream data, match the business term to be standardized with the terms in the cache, and if the matching is successful, replace the business term to be standardized with the corresponding standard term of the term in the cache; If the matching is unsuccessful, the business term to be standardized is matched with the pre-constructed business term graph, and if the matching is successful, the standard term is obtained, and the standard term is used to replace the business term to be standardized. If the matching is unsuccessful, the business term to be standardized is matched with the pre-constructed business term graph, and if the matching is successful, the standard term is obtained, and the standard term is used to replace the business term to be standardized.

[0092] In the specific implementation process, when receiving the structured request submitted by the user, the input data can be received through the RESTful API, and a load balancing mechanism is deployed to ensure stable response in a high-concurrency scenario. The core processing steps are as follows: 1) Input type judgment Based on Content-Type and binary characteristics, the request data type (text / voice / file) is accurately identified, and the routing basis is provided for subsequent differentiated processing.

[0093] The code for input type judgment is as follows: def detect_input_type(request): # Check Content-Type first if 'audio' in request.headers['Content-Type']: return 'voice' # Check file binary characteristics if request.files: file_header = request.files[0].read(4) if file_header.startswith(b'PK\x03\x04'): # ZIP format characteristics return 'excel' # Default text processing return 'text' 2) Input processing Convert the input into a standard text stream data to eliminate the influence of carrier differences on the downstream. The following table shows the conversion method for different data types.

[0094]

[0095] It should be noted that the above only provides one conversion method for text, voice and file, and in actual application, a suitable method can be selected for conversion according to actual needs, and the embodiments of the present application do not make specific limitations.

[0096] 3) Sensitive word filtering Real-time detection and interception of malicious content in user input to ensure system security and compliance.

[0097]

[0098] When an anomaly is detected, different measures are taken for different situations, Figure 5 The abnormality corresponding measures provided by the embodiments of the present application are shown in Figure 5 For example, if the risk level of the detected sensitive word is high, the attack IP is recorded and blocked, and then the security team is notified; if the risk level of the detected sensitive word is medium, a fuzzing error prompt is returned, and an audit log is recorded; if the risk level of the detected sensitive word is low, the sensitive word is replaced with a safe word with the corresponding meaning, and then the content is released, and the content review is marked.

[0099] 4) Term standardization service When no sensitive word anomaly is detected, the system converts non-standard expressions such as business slang, dialect, and abbreviation in user input into standard terms recognizable by the system in real time, ensuring consistency in downstream semantic understanding.

[0100] Figure 6 The term standardization method flowchart provided by the embodiments of the present application is shown in Figure 6 . 1. Cache priority query After receiving the term, the system first queries the multi-level cache to quickly return the standardization result of the high-frequency term: (1) L1 cache (local memory) An LRU cache implemented using Caffeine stores the mapping of recently accessed terms. If the term exists in the cache, the result is returned directly, otherwise the L2 cache is queried.

[0101] (2) L2 cache (Redis cluster) Store enterprise-level public terms to support cross-service sharing. Store business line differentiated terms through Hash structure. If the term exists in the cache, the result is returned directly, otherwise the term graph query is triggered.

[0102] 2. Term graph deep query For terms that do not hit the cache, query the business term graph constructed by Neo4j to match standard terms through the relationship network. Not only can the term nodes with completely consistent names or aliases be found, but also the search can be expanded along the SYNONYM (synonym) and ABBREVIATION (abbreviation) relationships.

[0103] Assume that the business term graph is a directed attribute graph , where the vertex set Business terms, data entities, business concepts; edge set Semantic relationship; attribute function Additional metadata.

[0104] It can be understood that the business term graph can also be updated, when a new term is detected The update process satisfies: Wherein, The similarity function between the term vector and the existing node is calculated by the graph neural network, The verification type constraint consistency function. By automatically updating the business term graph, the cost of manual maintenance can be reduced, and the response delay of new business terms is low.

[0105] 3. Fuzzy matching engine If the business term graph matching fails, start multi-strategy fuzzy matching: (1) Pinyin conversion Convert input to pinyin and match dialect or pronunciation similar terms.

[0106] (2) Edit distance Calculate the character difference between the input and the candidate term.

[0107] (3) Word vector similarity Calculate semantic similarity through a pre-trained word vector model (Word2Vec).

[0108] Finally, the scores of each strategy are integrated, and the final confidence is calculated by weighting. If it is in a high confidence interval, the graph is automatically updated and the standard term is returned, otherwise it will be included in the manual review queue, and the administrator will confirm the mapping relationship.

[0109] After standardizing the request data, context loading can be performed. The specific method is: first extract key metadata (such as session ID, user ID, department information), and trigger multi-level context parallel loading: session-level data (such as current filtering conditions, temporary variables) is quickly obtained from the Redis cluster, user-level data (such as personal preferences, historical query patterns) is loaded from PostgreSQL, and business-level data (such as data permissions, organizational structure) is pulled in real time through the enterprise resource planning ERP system API. The three types of context are intelligently merged in the fusion engine, giving priority to session-level data, while injecting business rules (such as data access permissions) and user habits (such as preferred chart types), generating an enhanced request object containing a complete analysis environment.

[0110] In another embodiment, Figure 7 The data conflict resolution method flowchart provided by the embodiment of the application is as shown in Figure 7As shown. Before merging, the system can detect whether there is a data conflict, where a data conflict refers to inconsistent data in multiple data sources, which is monitored by comparing the data of multiple data sources. If there is, a hierarchical degradation strategy is used to deal with the exception (mainly refers to switching to local cache when Redis fails), and the reading priority is session level > user level > business level: Session Level Description: Cache strategy for a single user session. When Redis fails, use the session-level local cache first.

[0111] Session Cache: When a user starts a session (e.g., logs in), cache relevant data (e.g., user information, permissions) in the local session memory.

[0112] User Level Description: Cache strategy for a specific user. If session-level cache is unavailable, switch to user-level cache to save the most important information.

[0113] User Cache: Maintain a ConcurrentHashMap in the application to store cache data for a specific user, such as user settings or preferences.

[0114] Fault Handling: When Redis is unavailable, check the user-level cache.

[0115] Business Level Description: If both session-level and user-level caches are unavailable, fall back to business-level cache. This is usually database-level data, such as statistical information, common configurations, etc.

[0116] Business Cache: Store some data in the application that is not frequently updated but frequently used (e.g., business logic settings, constant configurations) in local static cache or in-memory database.

[0117] The embodiment of the application converts the request data into text stream data, then matches the terms in the cache first, if the matching is unsuccessful, then matches the business term graph, and if the matching is still unsuccessful, then performs fuzzy matching. In the matching process, both matching efficiency and accuracy are taken into account.

[0118] On the basis of the above embodiment, the method further comprises: If a new business term is detected, extract a subgraph related to the new business term from the dynamic semantic graph; Calculate the relationship probability between the new business term and the business terms in the subgraph based on the topological structure features of the subgraph. generating a candidate triple according to a relationship probability; the candidate triple comprises the new business term, the business term in the subgraph, and the association relationship; calculating a confidence degree of the new business term based on the candidate triple, and adding the new business term to the dynamic semantic graph based on the candidate triple if the confidence degree is greater than a preset threshold.

[0119] In the specific implementation process, Figure 8 A dynamic semantic graph updating method provided by the embodiment of the application is shown in a flowchart as shown in Figure 8 . 1) New term detection technology The system receives multi-source data input, and then judges whether the data is structured data: if yes, it is matched through regular rules; if not, it is semantically encoded using the BERT model. Both processing methods will eventually generate a term fingerprint.

[0120] The system performs distributed sharding processing on the term fingerprint through local sensitive hashing (LSH), and realizes fuzzy matching based on the BK tree structure. In the matching process, it is judged whether it is a new term: if not, it is directly discarded; if it is a new term, its semantic dispersion (SD) is calculated, which is expressed as , wherein, represents the vector of the BERT embedding of the candidate term; represents the average-pooled BERT vector of the i-th context sentence; represents the number of context window sentences. When the SD value is greater than the critical value, the term is added to the candidate queue; otherwise, it is also discarded.

[0121] 2) Candidate term generation The ELECTRA model is used to encode the candidate term to obtain the context semantic representation, and then the GraphSAGE algorithm is used to sample related subgraphs from the knowledge graph, and the topological structure features of the term are extracted. Based on these features, the system predicts the potential relationship probability between terms: , wherein, represents the term feature vector; represents a set of predefined relationship types; represents the classification weight vector of the relationship r. Then and judge whether it is a structured term: if yes, it is converted to a standard form through regular rules; if not, a neural network is used to generate a relationship description. Both processing methods will eventually generate a candidate triple (entity-relation-entity).

[0122] The system preliminarily screens the confidence degree of the candidate triple, and outputs the triple that passes the screening to the evaluation module for subsequent processing, and directly discards the triple that does not pass the screening.

[0123] 3) Confidence evaluation model The candidate terms are subjected to multi-dimensional feature extraction, and then the optimized LightGBM model is used to predict the confidence (Conf) thereof, which is specifically represented as: wherein, represents a scalar of the first-order gradient of the loss function; represents a scalar of the second-order gradient of the loss function.

[0124] The system performs hierarchical processing according to the confidence score: when Conf ≥ 0.8, the term is automatically stored in the warehouse; when 0.5 ≤ Conf < 0.8, the term is transferred to the artificial review station for artificial review, and if the review is passed, it is stored in the warehouse, otherwise it is discarded; when Conf < 0.5, it is directly discarded. After the term is successfully stored in the warehouse, the system automatically updates the dynamic semantic atlas version and triggers the incremental training process of the model, realizing the continuous optimization and iterative update of the knowledge base.

[0125] The embodiments of the present application update the dynamic semantic atlas after detecting new terms, which facilitates subsequent accurate semantic analysis.

[0126] On the basis of the above-mentioned embodiments, the method further comprises: Each department calculates the model gradient based on local data; The model gradient is encrypted, and the encrypted model gradient is uploaded to the secure aggregation node; The encrypted model gradient of each department is fused by the secure aggregation node to obtain fused data; The semantic understanding model is updated based on the fused data; wherein the semantic understanding model is used to analyze the standardized data based on the pre-constructed dynamic semantic atlas to obtain an analysis result.

[0127] In the specific implementation process, in order to improve the performance of the semantic understanding model, the local data of each department (such as department A local data and department B local data) can be fused to drive the semantic understanding model to continuously iterate and break through the knowledge limitations of a single data source. Figure 9 The semantic understanding model optimization method provided by the embodiments of the present application is shown in Figure 9 .

[0128] Detailed and specific implementation of core technology path: In the process of gradient encryption transmission, each department processes the local data as follows: Local data preprocessing: Each department performs data preprocessing on the data set it holds, including cleaning, labeling, etc.

[0129] Local model training Each department uses its local data to train the model and calculates the corresponding gradient information.

[0130] Gradient encryption After calculating the gradient, each department will perform the following encryption on the gradient.

[0131] 1) Gradient encryption transmission The Paillier homomorphic encryption algorithm is used to realize the secure upload of gradients. Its core principle is to use the composite residue class mathematical problem to directly perform arithmetic operations on the gradient value in the encrypted state. The specific formula is as follows:

[0132] Where KeyGen represents the generation of a private key formula; pk represents the public key, sk represents the private key, and λ represents the least common multiple of two random prime numbers g and n. Enc represents the encryption formula, m represents the plaintext to be encrypted, r represents a random number, g and n are random prime numbers for generating secret keys. Dec represents the decryption formula, L(u) = (u-1) / n, which helps extract information related to the random prime number n from u to recover the plaintext. c represents the ciphertext, and g represents the plaintext.

[0133] In specific implementation, assuming there are two participants, department A and department B, each participant generates a public-private key pair, and after quantizing the local model gradient value to a fixed-point number, the public key is used to encrypt each dimension of the gradient vector. The encrypted gradient forms a ciphertext vector, ensuring that the aggregation server cannot decrypt the original gradient of a single participant, but can calculate the total gradient through the homomorphic property. This scheme satisfies the additive homomorphism, that is, the multiplication of encrypted gradients is equivalent to the addition of plaintext gradients, and through the large integer factorization problem, it is ensured that even if the ciphertext is obtained, the original data cannot be deduced in reverse.

[0134] 2) Secure aggregation protocol Based on secure multi-party computation (SMPC), a distributed aggregation protocol is designed to realize gradient fusion in an encrypted state through algebraic homomorphism, which is represented as:

[0135] Where, represents the ciphertext of node i, K represents the total number of nodes, Enc represents the Paillier homomorphic encryption algorithm mentioned above, represents the plaintext segment of the i-th node.

[0136] The encrypted gradients uploaded by participants are element-wise multiplied at the aggregation node, which mathematically completes the gradient summation without exposing individual information. The protocol introduces a double-mask technique to cope with the dynamic exit scenario of participants: a random mask is added before each gradient upload, and the mask effect is eliminated by a negotiated offset factor during aggregation. This process ensures that the valid aggregation result can only be restored when a sufficient number of participants collaborate, and neither a single participant nor the aggregation server can obtain the private information of others.

[0137] fn secure_aggregate(gradients: Vec <ciphertext>) ->Ciphertext { let mut sum = Ciphertext::one(); for g in gradients { sum = sum * g; / / Homomorphic multiplication is addition } sum = sum + gaussian_noise(epsilon); / / Differential noise sum } The embodiments of the application fuse the encrypted model gradients of each department through the secure aggregation node, and update the semantic understanding model based on the fused data, thereby improving the accuracy of subsequent semantic understanding model analysis.

[0138] On the basis of the above-embodiment, the method further comprises: After listening to the metadata change event of the business system, determining the affected model based on the metadata change event and the pre-constructed blood relation dependency graph; If the semantic understanding module is the affected model, retraining the semantic understanding module; If the logic compilation module is the affected model, reconstruct the executable operation chain.

[0139] In the specific implementation process, Figure 10 The system update method flowchart provided by the embodiments of the application is shown in Figure 10 The business intelligent system listens to its own metadata change event, and when the change is detected, triggers the impact analysis engine immediately, determines the affected range by traversing the dependency relationship in the field knowledge graph. According to the model type of the change impact, branch processing is performed: if the semantic understanding module is involved, the model retraining process is started, and the elastic weight solidification (EWC) technology is used to realize the model hot update; if the logic compilation module is affected, the corresponding operation chain is automatically reconstructed and the logic consistency is verified. After completing the update, a new version of the model or the operation chain is generated, which is verified by the version control and enters the gray release stage, first verifies the service for a small part of user requests, confirms that there is no error, and then goes online, and at the same time, the change information is synchronized to the monitoring system to form a closed-loop management.

[0140] Detailed specific implementation of core technology path: 1) Event listening and capturing We monitor metadata change events in the business system in real time using database log parsing tools, and identify key DDL operations (CREATE / ALTER / DROP) using regular expression pattern matching. When a change in table structure or field definition is detected, the event is standardized into a unified format and published to a message queue. The event content includes the change type, affected objects, and detailed operation statements, providing structured input for subsequent analysis.

[0141] class ChangeDetector: def __init__(self): self.cdc_connectors = { 'MySQL': DebeziumConnector(config), 'Oracle': LogMinerConnector(config) } def listen(self): while True: change = self.poll_from_sources() # Polling of change events from multiple sources if self.is_metadata_change(change): # Regular expression-based pattern matching self.publish_to_kafka(change) 2) Scope of Impact Analysis Based on a pre-constructed bloodline dependency graph In this graph, V represents a node and E represents a directed edge; the lineage dependency graph is constructed based on the dependencies between data. For example, if the employee access log table depends on the employee information table (i.e., the upstream table field originates from the downstream table), then there is a lineage dependency relationship. An improved breadth-first search algorithm is used to traverse the nodes affected by the change: def propagate_impact(change_node): impacted = set() queue = deque([change_node]) while queue: node = queue.popleft() for neighbor in graph.neighbors(node): if neighbor not in impacted: impacted.add(neighbor) queue.append(neighbor) return impacted Meanwhile, the algorithm comprehensively considers the topological importance and business criticality of nodes in the graph, calculates the impact score of each affected entity, and generates a priority processing queue accordingly.

[0142] Impact strength calculation: where Depth represents the shortest path length from the node to v; MaxDepth represents the maximum depth of the graph; PageRank represents the PageRank (business criticality of v) value of node v, ranging from [0, 1].

[0143] Change priority sorting: where represents the urgency weight, ranging from [0, 1], determined by the change type.

[0144] And for non-destructive changes such as adding fields, only mark the direct dependent objects; for destructive changes such as deleting fields, recursively track all indirect dependencies.

[0145] 3) Model dynamic update According to the model type of the change impact, branch processing is performed: 1) Involve semantic understanding module, then start model retraining process, use elastic weight consolidation (EWC) technology to realize model hot update, when fine-tuning model adapts to new metadata, and through Fisher information matrix constraint important parameter adjustment amplitude, prevent catastrophic forgetting.

[0146] Elastic weight consolidation (EWC) loss function: where represents the regularization strength; represents the old model parameter; represents the Fisher information matrix diagonal element, calculated as: .

[0147] 2) For logic compilation module, then apply different reconstruction strategies based on change type: field increase and decrease trigger data binding update; table structure change rewrite associated query; calculation logic change replace formula tree node. All modifications are guaranteed to be grammatically correct through incremental recompilation of AST.

[0148] 1. Field increase and decrease processing strategy When detecting the increase or decrease of table structure field, the system automatically scans the field reference in the existing operation chain, uses dynamic binding algorithm to perform data binding update: public void updateBindings(TableSchema newSchema) { / / New and old schema difference analysis Set <string>addedColumns = Sets.difference(newSchema.columns(),oldSchema.columns()); Set <string>removedColumns = Sets.difference(oldSchema.columns(),newSchema.columns()); / / Update column references in the operation chain for (Operation op : opChain) { if (op instanceof ColumnRef) { ColumnRef ref = (ColumnRef)op; if (removedColumns.contains(ref.columnName())) { ref.markDeprecated(); / / Mark for replacement } } } / / Automatically complete new field references if (!addedColumns.isEmpty()) { injectDefaultMappings(addedColumns); } } 2. Table structure change handling When a significant table structure change occurs, the system initiates the JOIN rewrite engine: -- Original query SELECT o.*, p.price FROM orders o JOIN products p ON o.pid = p.id -- After products table is split into product_base and product_inventory SELECT o.*, b.price, i.stock FROM orders o JOIN product_base b ON o.pid = b.id JOIN product_inventory i ON o.pid = i.product_id 3. Calculation logic change handling When a metric calculation logic change is detected, trigger formula tree replacement: public class FormulaTreeReplacer { / / Main entry method public ASTNode replace(ASTNode root, ChangeSpec change) { / / 1. Locate the change node ASTNode target = findTargetNode(root, change.getOldPattern()); / / 2. Validate the replacement safety validateReplacement(target, change.getNewNode()); / / 3. Perform the replacement return performReplacement(root, target, change.getNewNode()); } private ASTNode findTargetNode(ASTNode root, ASTNode pattern) { / / Use tree traversal algorithm to locate the target node Stack <astnode>stack = new Stack<>(); stack.push(root); while (!stack.isEmpty()) { ASTNode current = stack.pop(); if (matchNode(current, pattern)) { return current; } for (ASTNode child : current.getChildren()) { stack.push(child); } } return null; } private boolean matchNode(ASTNode node, ASTNode pattern) { / / Implement node matching logic (simplified version) if (node.getNodeType() != pattern.getNodeType()) { return false; } if (node ​​instanceof AggregateNode&&pattern instanceof AggregateNode){ return ((AggregateNode) node).getFuncType() == ((AggregateNode) pattern).getFuncType(); } / / Matching rules for other node types... return true; } private void validateReplacement(ASTNode oldNode, ASTNode newNode) { / / Type compatibility check if (!isTypeCompatible(oldNode.getDataType(), newNode.getDataType())){ throw new TypeMismatchException( "Incompatible types: " + oldNode.getDataType() + " vs " +newNode.getDataType()); } / / side effect analysis Set <nodetype>impactScope = analyzeImpact(oldNode); if (impactScope.contains(NodeType.HAVING)) { adjustThresholds(newNode); } } private ASTNode performReplacement(ASTNode root, ASTNode oldNode,ASTNode newNode) { / / Record the original parent node reference ASTNode parent = oldNode.getParent(); / / Perform replacement if (parent != null) { parent.replaceChild(oldNode, newNode); } else { root = newNode; / / Replace the root node } / / Type re-inference propagateTypes(newNode); return root; } / / Helper methods private boolean isTypeCompatible(DataType oldType, DataType newType){ / / Type compatibility rule matrix Map <DataType, Set <datatype>> compatibility = Map.of( DataType.INT, Set.of(DataType.INT, DataType.FLOAT, DataType.DECIMAL), DataType.FLOAT, Set.of(DataType.FLOAT, DataType.DECIMAL), / / Other type rules... ); return compatibility.getOrDefault(oldType, Set.of()).contains(newType); } } 4. AST incremental recompilation All changes are finally guaranteed to be syntactically correct through incremental compilation: def rewriteAST(original: ASTNode): ASTNode = original match { case Select(columns, From(tables), Where(cond)) => val newTables = rewriteTables(tables) val newColumns = rewriteColumns(columns) Select(newColumns, From(newTables), Where(cond)) case _ =>super.rewrite(original) } It should be noted that the code provided in the above embodiments is only an example, and any code that can implement the corresponding functions in the above embodiments can be used, and the embodiments of the present application do not limit the implementation code of each function.

[0149] The embodiments of the present application automatically trigger the reconstruction and verification of the business intelligent system after listening to the metadata change event of the business system, so that the analysis logic and the business rules are synchronized in real time.

[0150] Figure 11 A service intelligent system structure schematic diagram provided by the embodiment of the application, the device can be a module, a program segment or code on an electronic device. It should be understood that the system and the above-mentioned Figure 1 The method embodiment corresponds, and can execute Figure 1 The device corresponds to each step involved in the method embodiment, and the specific functions can be referred to the description in the above, and the detailed description is appropriately omitted here to avoid repetition. The system comprises a semantic standardization module 1101, a semantic understanding module 1102, a logic compilation module 1103 and a physical execution module 1104, wherein: The semantic standardization module 1101 is configured to receive request data, and perform standardization processing on the request data to obtain standardized data; The semantic understanding module 1102 is configured to analyze the standardized data based on a pre-constructed dynamic semantic graph to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities and business concepts; The logic compilation module 1103 is configured to perform logic compilation on the analysis result to generate an executable operation chain; the executable operation chain comprises sequential atomic operation steps, and the atomic operation steps comprise data processing logic and metadata binding information; The physical execution module 1104 is configured to run the executable operation chain to obtain an analysis result corresponding to the request data.

[0151] On the basis of the above-mentioned embodiment, the semantic understanding module 1102 is specifically configured to: match the standardized data with the dynamic semantic graph, and if the matching is successful, extract a target semantic structure matched successfully from the dynamic semantic graph; generate a structured analysis tree based on the target semantic structure; perform intent classification and entity link analysis based on the structured analysis tree to obtain an intent category and a standardized entity; perform semantic relationship construction according to the intent category and the standardized entity to obtain the analysis result.

[0152] On the basis of the above-mentioned embodiment, the logic compilation module 1103 is specifically configured to: extract semantic elements from the analysis result, convert the semantic elements into a relational algebra expression, and generate an abstract syntax tree based on the relational algebra expression; convert the abstract syntax tree into a logic plan expression; generate an optimization space according to the logic plan expression, and generate a cost function according to the abstract syntax tree; perform dynamic programming according to the optimization space and the cost function to obtain an optimal plan; mapping logical operation methods in the optimal plan to physical operation types and generating a directed acyclic graph based on data dependency relationships in the optimal plan; converting the physical operation types into executable code; and generating the executable operation chain based on the executable code and the directed acyclic graph.

[0153] On the basis of the above-mentioned embodiments, the physical execution module 1104 is specifically configured to: allocating an execution engine for an atomic operation step in the executable operation chain through intelligent routing; processing the corresponding principle operation step through the execution engine to obtain a processing result; fusing the processing result of the execution engine to obtain the analysis result.

[0154] On the basis of the above-mentioned embodiments, the semantic standardization module 1101 is specifically configured to: identifying the data type of the request data, and converting the request data into text stream data based on the data type; extracting a to-be-standardized business term in the text stream data, matching the to-be-standardized business term with a term in the cache, and if the matching is successful, replacing the to-be-standardized business term with a standard term corresponding to the term in the cache; if the matching is unsuccessful, matching the to-be-standardized business term with a pre-constructed business term graph, and if the matching is successful, obtaining a standard term and replacing the to-be-standardized business term with the standard term; if the matching is unsuccessful, performing fuzzy matching on the to-be-standardized business term.

[0155] On the basis of the above-mentioned embodiments, the system further comprises a domain knowledge dynamic extraction module, configured to: if a new business term is detected, extracting a subgraph related to the new business term from a dynamic semantic graph; calculating a relationship probability between the new business term and a business term in the subgraph based on a topological structure feature of the subgraph; generating a candidate triple according to the relationship probability; the candidate triple comprises the new business term, the business term in the subgraph, and an association relationship; calculating a confidence degree of the new business term based on the candidate triple, and if the confidence degree is greater than a preset threshold, adding the new business term to the dynamic semantic graph based on the candidate triple.

[0156] On the basis of the above-mentioned embodiments, the system further comprises a federal knowledge evolution module, configured to: Each department calculates a model gradient based on local data respectively; The model gradient is encrypted, and the encrypted model gradient is uploaded to a secure aggregation node; The encrypted model gradient of each department is fused by the secure aggregation node to obtain fused data; The semantic understanding model is updated based on the fused data; wherein the semantic understanding model is used to analyze the standardized data based on a pre-constructed dynamic semantic graph to obtain an analysis result.

[0157] On the basis of the above embodiment, the system further comprises a model real-time collaboration module, configured to: After listening to the metadata change event of the business system, determine the affected model based on the metadata change event and the pre-constructed blood relationship dependency graph; If the semantic understanding module is the affected model, retrain the semantic understanding module; If the logic compilation module is the affected model, reconstruct the executable operation chain.

[0158] Figure 12 The electronic device entity structure schematic diagram provided by the embodiment of the present application is shown in Figure 12 The electronic device comprises a processor 1201, a memory 1202 and a bus 1203; wherein: The processor 1201 and the memory 1202 complete mutual communication through the bus 1203; The processor 1201 is configured to call the program instructions in the memory 1202 to execute the method provided by each method embodiment, for example, comprising: receiving request data, standardizing the request data to obtain standardized data; based on a pre-constructed dynamic semantic graph, the standardized data is analyzed to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terminology, data entity and business concept; the analysis result is logically compiled to generate an executable operation chain; the executable operation chain comprises sequential atomic operation steps, and the atomic operation steps contain data processing logic and metadata binding information; running the executable operation chain, obtaining the analysis result corresponding to the request data.

[0159] The processor 1201 can be an integrated circuit chip having a signal processing capability. The processor 1201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; or can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The processor 1201 can implement or execute the various methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor.

[0160] The memory 1202 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0161] The embodiment discloses a computer program product, the computer program product includes a computer program stored on a non-transitory computer readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the method provided by each method embodiment, for example, including: receiving request data, standardizing the request data to obtain standardized data; based on a pre-constructed dynamic semantic graph, the standardized data is analyzed to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities, and business concepts; the analysis result is logically compiled to generate an executable operation chain; the executable operation chain includes sequential atomic operation steps, the atomic operation steps include data processing logic and metadata binding information; running the executable operation chain, obtaining an analysis result corresponding to the request data.

[0162] The embodiment provides a non-transitory computer readable storage medium storing computer instructions, the computer instructions causing the computer to execute the method provided in each method embodiment, for example comprising: receiving request data, performing standardization processing on the request data to obtain standardized data; performing analysis on the standardized data based on a pre-constructed dynamic semantic graph to obtain an analysis result; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities and business concepts; performing logical compilation on the analysis result to generate an executable operation chain; the executable operation chain comprises sequential atomic operation steps, and the atomic operation steps comprise data processing logic and metadata binding information; running the executable operation chain to obtain an analysis result corresponding to the request data.

[0163] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The embodiment described above is only schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, and electrical, mechanical or other forms.

[0164] In addition, the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments.

[0165] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0166] In this paper, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations.

[0167] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.< / datatype> < / nodetype> < / astnode> < / string> < / string> < / ciphertext>

Claims

1. A data processing method based on a business intelligence system, characterized in that, include: Receive request data, standardize the request data, and obtain standardized data; The standardized data is parsed based on a pre-constructed dynamic semantic graph to obtain the parsing results; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities, and business concepts. The parsing results are logically compiled to generate an executable operation chain; the executable operation chain includes sequential atomic operation steps, and the atomic operation steps contain data processing logic and metadata binding information; Run the executable operation chain to obtain the analysis results corresponding to the requested data.

2. The method according to claim 1, characterized in that, The process of parsing the standardized data based on a pre-constructed dynamic semantic graph to obtain the parsing results includes: The standardized data is matched with the dynamic semantic graph. If the match is successful, the target semantic structure that has been successfully matched is extracted from the dynamic semantic graph. A structured parse tree is generated based on the target semantic structure; Based on the structured parse tree, intent classification and entity link analysis are performed to obtain intent categories and standardized entities; The semantic relationship is constructed based on the intent category and the standardized entity to obtain the parsing result.

3. The method according to claim 1, characterized in that, The step of logically compiling the parsed results to generate an executable operation chain includes: Semantic elements are extracted from the parsing results, the semantic elements are converted into relational algebra expressions, and an abstract syntax tree is generated based on the relational algebra expressions; Convert the abstract syntax tree into a logical plan expression; An optimization space is generated based on the logical plan expression, and a cost function is generated based on the abstract syntax tree. Dynamic programming is performed based on the optimization space and the cost function to obtain the optimal plan; The logical operation algorithms in the optimal plan are mapped to physical operation types, and a directed acyclic graph is generated based on the data dependencies in the optimal plan. Convert the physical operation type into executable code; The executable operation chain is generated based on the executable code and the directed acyclic graph.

4. The method according to claim 1, characterized in that, The process of running the executable operation chain to obtain the analysis results corresponding to the requested data includes: Execution engines are assigned to atomic operation steps in the executable operation chain via intelligent routing; The execution engine processes the corresponding principle operation steps to obtain the processing result; The processing results of the execution engine are fused to obtain the analysis results.

5. The method according to claim 1, characterized in that, The standardization process for the request data includes: Identify the data type of the request data, and convert the request data into text stream data based on the data type; Extract the business terms to be standardized from the text stream data, match the business terms to be standardized with the terms in the cache, and if the match is successful, replace the business terms to be standardized with the standard terms corresponding to the terms in the cache. If the matching fails, the business term to be standardized is matched with the pre-built business term map. If the matching succeeds, the standard term is obtained and the business term to be standardized is replaced by the standard term. If the matching fails, a fuzzy matching will be performed on the business terms to be standardized.

6. The method according to claim 1, characterized in that, The method further includes: If a new business term is detected, a subgraph related to the new business term is extracted from the dynamic semantic graph; Calculate the probability of the relationship between the new business term and the business terms in the subgraph based on the topological features of the subgraph; Candidate triples are generated based on the relationship probabilities; the candidate triples include the new business terms, business terms in the subgraph, and the association relationships. The confidence level of the new business term is calculated based on the candidate triples. If the confidence level is greater than a preset threshold, the new business term is added to the dynamic semantic graph based on the candidate triples.

7. The method according to claim 1, characterized in that, The method further includes: Each department calculates the model gradient based on its local data; The model gradient is encrypted, and the encrypted model gradient is uploaded to the secure aggregation node; The encrypted model gradients of each department are fused through the secure aggregation node to obtain fused data; The semantic understanding model is updated based on the fused data; wherein the semantic understanding model is used to parse the standardized data based on a pre-constructed dynamic semantic graph to obtain the parsing results.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: After monitoring metadata change events in the business system, the affected models are determined based on the metadata change events and the pre-built lineage dependency graph. If the semantic understanding module is an affected model, then the semantic understanding module is retrained; If the logic compilation module is the affected model, then the executable operation chain is refactored.

9. A business intelligence system, characterized in that, include: The semantic standardization module is used to receive request data, standardize the request data, and obtain standardized data. The semantic understanding module is used to parse the standardized data based on a pre-built dynamic semantic graph to obtain the parsing results; the dynamic semantic graph is a directed attribute graph constructed based on business terms, data entities, and business concepts. A logic compilation module is used to logically compile the parsed results to generate an executable operation chain; the executable operation chain includes sequential atomic operation steps, and the atomic operation steps contain data processing logic and metadata binding information; The physical execution module is used to run the executable operation chain and obtain the analysis results corresponding to the requested data.

10. An electronic device, characterized in that, include: Processor, memory, and bus, among which: The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-8 by calling the program instructions.

11. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-8.

Images