Wide table data analysis method and system based on semantic association

Abstract

The invention discloses a wide table data analysis method and system based on semantic association, and the method comprises the steps: receiving a natural language query demand input by a user, carrying out the semantic understanding through a pre-training language model combined with a financial knowledge graph, and generating a corresponding semantic expression; performing semantic analysis on the semantic expression, and disassembling the semantic expression into a logic expression containing a query demand and an analysis demand; based on the logic expression of the query demand, matching a corresponding wide table and field information through a preset index library, generating an executable query statement, and obtaining target data from the wide table; and according to the logic expression of the analysis demand, generating a corresponding data analysis code by using the code generation model, and analyzing the target data to generate an analysis result. The limitation of a traditional wide table fixed association mode is broken through, efficient integration and intelligent analysis of wide table data are achieved, the flexibility and accuracy of data processing are improved, and the limitation of wide table application in a fixed scene is broken through.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and in particular to a wide table data analysis method and system based on semantic association. Background Technology

[0002] With the advent of the big data era, enterprises and institutions have accumulated massive amounts of data. Wide tables, as a data storage method, improve query performance by integrating multiple business-related metrics, dimensions, and attributes into a single table. However, wide tables also have significant drawbacks: high data redundancy, high storage costs, and traditional query methods rely on fixed-structure SQL statements, lacking flexibility. Users typically need professional data querying knowledge (such as SQL writing skills) to effectively utilize wide table data, limiting its use by non-technical users. In existing technologies, natural language processing has been applied to data querying, for example, generating SQL queries from natural language. However, these methods have limitations in complex scenarios; especially in the financial sector, financial data and certain technical terms have highly specialized, security, and compliance requirements. Using existing general data query methods can easily lead to ambiguity, and users' queries using natural language may contain vague expressions or intentions, making it difficult to accurately understand specific query and analysis needs, resulting in inaccurate final analysis results. Therefore, there is an urgent need for an intelligent wide table data analysis method that can understand natural language queries, accurately parse user intent, generate reliable executable code, and ensure data security and compliance.

[0003] The "Big Data Analysis Method for Power Grid Operation Data Based on Business Wide Tables," published in Chinese patent literature (CN114153847A, published on March 8, 2022), includes: parsing business systems and data tables to generate a data analysis service catalog; selecting data tables from multiple business systems from the data analysis service catalog as needed; configuring wide tables; determining whether wide tables can be formed based on the relationships between data tables, and generating wide table descriptions if relationships are found; configuring big data applications for customized wide tables; acquiring business data from a data warehouse to form wide tables; cleaning the business data; and performing big data analysis and result display. This technology uses a WYSIWYG approach to configure associated data tables, data fields, and relationships in power wide tables, automatically generating views based on the configuration results, automatically acquiring data from the data warehouse, and then using big data analysis tools to conduct big data analysis and uncover data value. However, this technology only provides a simple solution for retrieving data from wide tables, which cannot meet the high requirements of professionalism and security in complex scenarios; it is difficult to accurately understand the intent of users' natural language query requests, and lacks accurate understanding and processing of professional domain knowledge, resulting in inaccurate query analysis results. Summary of the Invention

[0004] This invention aims to overcome the problems of existing wide-table data analysis methods being unable to adapt to the high requirements of professionalism and security in complex scenarios, struggling to accurately understand the intent of users' natural language query requests, and lacking accurate understanding and processing of professional domain knowledge, resulting in inaccurate query analysis results. It provides a wide-table data analysis method and system based on semantic association.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A wide-table data analysis method based on semantic association includes: It receives natural language query requests from users, uses a pre-trained language model combined with financial knowledge graphs for semantic understanding, and generates corresponding semantic representations. Semantic parsing is performed on the semantic representation, breaking it down into logical expressions that include query requirements and analysis requirements; Based on the logical expression of the query requirements, the corresponding wide table and field information are matched through the preset indicator library to generate an executable query statement and obtain the target data from the wide table. Based on the logical expression of the analysis requirements, the code generation model is used to generate corresponding data analysis code, which is then used to analyze the target data and generate analysis results.

[0006] This invention utilizes a natural language interface for data querying, effectively lowering the technical barrier to entry and enabling business personnel without technical backgrounds to directly perform data retrieval and analysis, significantly improving query efficiency. By introducing a pre-trained language model enhanced with financial knowledge graphs, the system can deeply understand user query intent, ensuring accurate matching of semantic parsing and business logic, and greatly improving the accuracy of query and analysis results. A layered parsing strategy is adopted, decomposing natural language expressions into structured query and analysis requirements, and dynamically generating suitable query statements or analysis programs using a code generation mechanism, giving the system flexibility and adaptability to handle complex and ever-changing analysis scenarios. By establishing a multi-level code verification mechanism, the generated code undergoes syntax, logic, and boundary condition checks, effectively ensuring code quality and the reliability of execution results. The system breaks through the limitations of traditional wide-table fixed association patterns, achieving efficient integration and intelligent analysis of wide-table data, improving data processing flexibility and accuracy while overcoming the limitations of wide-table applications in fixed scenarios.

[0007] Preferably, the semantic parsing of the semantic expression, breaking it down into logical expressions that include query requirements and analysis requirements, includes: Combinatorial category grammar is used to perform preliminary analysis of semantic expressions, identifying key financial concepts and sentence structures; The initial parsing results are input into a deep parsing unit based on a neural network to capture the dependencies between semantic units and generate structured logical expressions.

[0008] Preferably, the step of generating corresponding data analysis code using a code generation model includes: The code generation model is a sequence-to-sequence model, which includes an encoder that encodes the analysis requirement description into a fixed-length vector representation, and a decoder that generates a corresponding code sequence based on the vector representation.

[0009] Preferably, the process of obtaining the pre-trained language model combined with the financial knowledge graph includes: Financial entities and relationships between entities are extracted from multi-source financial data to construct a financial knowledge graph. The financial entities include at least customers, accounts, products, risk events, and policies and regulations. The financial knowledge graph is injected into a pre-trained language model and trained to obtain a pre-trained language model that incorporates the financial knowledge graph.

[0010] Preferably, the construction of the financial knowledge graph includes: Collect and preprocess multi-source financial data, clean up null and outlier values ​​and standardize field formats for structured data, and extract key information from unstructured data using financial NLP tools. A hybrid extraction scheme combining rules and machine learning is used to extract entities and relationships, generating a structured knowledge graph.

[0011] Preferably, the preset indicator library includes at least the wide table name, the field names within the wide table, synonyms and abbreviations of the indicator names.

[0012] Preferably, after generating the corresponding data analysis code using the code generation model, a verification step is also included: The generated data analysis code is validated, including syntax checking, functional logic verification, and data type compliance checking. If the validation fails, an error message is sent back to regenerate the data analysis code until the validation passes.

[0013] A wide-table data analysis system based on semantic association includes: The indicator library construction module builds and stores a preset indicator library. The natural language processing module receives and understands the user's natural language query requirements and generates corresponding semantic representations. The semantic parsing module breaks down semantic expressions into logical expressions that include query and analysis requirements; The query execution module generates and executes query statements based on the logical expression of the query requirements to obtain wide table data; The analysis and execution module generates and executes data analysis code based on the logical expression of the analysis requirements, and obtains the analysis results.

[0014] Preferably, the wide table data analysis system further includes: The knowledge graph unit, set up in the natural language processing module, transforms diverse financial data into a structured knowledge graph; The code verification module verifies and corrects the generated data analysis code.

[0015] Preferably, the semantic parsing module includes: The preliminary parsing unit performs preliminary analysis of semantic expressions based on combinatorial category grammar, identifying key financial concepts and sentence structures; The deep parsing unit captures the dependencies between semantic units based on the preliminary parsing results and generates structured logical expressions.

[0016] This invention offers the following advantages: Through a natural language interface, users can perform data queries without needing to master complex SQL syntax, improving query efficiency and significantly lowering the barrier to entry; the pre-trained language model, combined with a financial knowledge graph, accurately understands the user's query intent, ensuring the accuracy of queries and analysis; the hierarchical parsing strategy decomposes semantic representations into logical expressions containing query and analysis requirements, and, combined with a code generation mechanism, generates corresponding query or analysis code, adapting to various complex query and analysis needs; the code verification mechanism ensures the quality of generated code and the reliability of analysis results; it breaks the limitations of traditional wide-table fixed association patterns, achieving efficient integration and intelligent analysis of wide-table data, improving the flexibility and accuracy of data processing, and overcoming the limitations of wide-table applications in fixed scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart of a wide table data analysis method based on semantic association in this invention.

[0018] Figure 2 This is another flowchart of a wide table data analysis system based on semantic association in this invention. Detailed Implementation

[0019] The present invention will now be further described with reference to the accompanying drawings and specific embodiments.

[0020] like Figure 1 As shown, a wide table data analysis method based on semantic association includes: It receives natural language query requests from users, uses a pre-trained language model combined with financial knowledge graphs for semantic understanding, and generates corresponding semantic representations. Semantic parsing is performed on the semantic representation, breaking it down into logical expressions that include query requirements and analysis requirements; Based on the logical expression of the query requirements, the corresponding wide table and field information are matched through the preset indicator library to generate an executable query statement and obtain the target data from the wide table. Based on the logical expression of the analysis requirements, the code generation model is used to generate corresponding data analysis code, which is then used to analyze the target data and generate analysis results.

[0021] This invention utilizes a natural language interface for data querying, effectively lowering the technical barrier to entry and enabling business personnel without technical backgrounds to directly perform data retrieval and analysis, significantly improving query efficiency. By introducing a pre-trained language model enhanced with financial knowledge graphs, the system can deeply understand user query intent, ensuring accurate matching of semantic parsing and business logic, and greatly improving the accuracy of query and analysis results. A layered parsing strategy is adopted, decomposing natural language expressions into structured query and analysis requirements, and dynamically generating suitable query statements or analysis programs using a code generation mechanism, giving the system flexibility and adaptability to handle complex and ever-changing analysis scenarios. By establishing a multi-level code verification mechanism, the generated code undergoes syntax, logic, and boundary condition checks, effectively ensuring code quality and the reliability of execution results. The system breaks through the limitations of traditional wide-table fixed association patterns, achieving efficient integration and intelligent analysis of wide-table data, improving data processing flexibility and accuracy while overcoming the limitations of wide-table applications in fixed scenarios.

[0022] Specifically, the pre-defined indicator library should include at least the wide table name, the field names within the wide table, and synonyms and abbreviations for the indicator names. The pre-defined indicator library is constructed by compiling the wide table names, field names within the wide table, commonly used colloquial terms, abbreviations, and synonyms related to the indicator names from various business scenarios into an indicator set.

[0023] When a user has a data query requirement, the query statement entered by the user is semantically understood by a pre-trained language model (such as BERT, GPT, etc.) of natural language processing. Natural language is converted into a semantic representation that the computer can understand, keyword information is extracted, and at the same time, based on the context awareness capability of the model, the context information in the query statement is captured to more accurately understand the user's intent.

[0024] Semantic parsing technology transforms extracted user demand features into logical structures, clarifies the relationships between various semantic units, and breaks down one or more query conditions. For example, in the query "Find young customers who have made more than 5 online purchases in the past month and have an account balance greater than 10,000 yuan", the semantic parser will identify conditions such as "in the past month", "more than 5 online purchases", "account balance greater than 10,000 yuan", and "young customers", and analyze the logical relationships between them (such as "AND" relationships). It will then transform these relationships into logical expressions with clear query objects, query conditions, and analytical requirements, such as "Query the data in [table name] under [conditions]".

[0025] The system retrieves the required wide table and related wide table fields from a pre-defined indicator library based on semantic matching, and fills these fields into the corresponding positions in the logical expression to form a complete SQL statement for querying. The generated SQL statement is then executed to query the wide table data, generating a CSV file which is returned to the wide table data analysis system.

[0026] Based on the semantic parsing results, we understand the user's data analysis needs (such as statistics, prediction, classification, etc.). Using a code generation model based on a sequence-to-sequence (Seq2Seq) model, we generate corresponding Python code sequences to meet these needs. The model's attention mechanism allows the decoder to focus more on the parts of the encoder output relevant to the current generation step when generating code. Executing the generated Python code sequences yields the data analysis results, which are then visualized.

[0027] As a specific implementation, semantic parsing of the semantic expression is performed, breaking it down into logical expressions that include query requirements and analysis requirements, including: Combinatorial category grammar is used to perform preliminary analysis of semantic expressions, identifying key financial concepts and sentence structures; The initial parsing results are input into a deep parsing unit based on a neural network to capture the dependencies between semantic units and generate structured logical expressions.

[0028] Specifically, the first step is to introduce a rule-based preliminary parsing module. This module uses predefined grammatical rules and financial terminology to perform preliminary processing on the input natural language text using combinatorial category grammar. It quickly identifies key financial concepts in the text and analyzes the basic structure of sentences to determine key components such as subject, verb, and object.

[0029] For example, Combinatorial Category Grammar (CCG) can be used to quickly identify key financial concepts in text, such as "stocks," "bonds," and "interest rates," and to preliminarily analyze the basic structure of sentences, identifying key components such as subject, verb, and object. This step provides a basic framework and key information for subsequent parsing, significantly narrowing the processing scope of neural network models and improving parsing efficiency. Simultaneously, rule-based preliminary parsing can, to a certain extent, ensure the accuracy and interpretability of the parsing results. For natural language expressions with clear rules and semantic clarity, reliable parsing results can be directly provided.

[0030] The initial parsing results are then input into a neural network-based deep parsing module. Leveraging its powerful self-learning and feature extraction capabilities, the neural network model performs in-depth analysis of the pre-processed text. A Long Short-Term Memory (LSTM) network is used to process the text sequence information, capturing long-term dependencies between words. An attention mechanism is combined to calculate the relevance of each part of the text to the current parsing task, assigning different weights to different parts. This allows the model to focus more on important information during parsing, thereby achieving deep semantic analysis and logical relationship recognition of the text, generating accurate logical representations.

[0031] In semantic parsing, LSTM can accurately understand the semantics of the current word based on the contextual information, thus better completing the semantic parsing task. For example, for the sentence "Apple's stock price has risen by 20% in the past year, thanks to the successful launch of its new products," LSTM can comprehensively consider the context of the entire sentence and accurately understand the semantic relationships between words such as "Apple," "stock price," "rise," and "new product launch," thereby generating an accurate logical form. Simultaneously, combined with attention mechanisms, neural network models can focus more intently on key information in the text, further improving the accuracy of semantic parsing.

[0032] The specific implementation of the Long Short-Term Memory (LSTM) network combined with the attention mechanism is as follows: a bidirectional LSTM network is used to encode the initial parsing results of the input, obtaining the forward and backward contextual information of each word; on this basis, a multi-head attention mechanism is introduced, enabling the model to simultaneously pay attention to information from different representation subspaces. By calculating the relevance scores between the query vector, key vector, and value vector, the encoder output is weighted and summed, thereby dynamically focusing on the semantic unit most relevant to the current generation step during the parsing process. This mechanism is particularly suitable for processing long text sequences with complex financial contexts, effectively solving the problems of referential resolution and semantic ambiguity, and improving the parsing accuracy of financial professional texts.

[0033] After completing the initial and deep semantic parsing, the task decomposition engine in the existing NL2SQL technology takes over the deeply parsed semantic representation and breaks it down into multiple atomic tasks. For each atomic task, a corresponding SQL query statement or analysis instruction is generated, thereby achieving the goal of decomposing the semantic parsing requirements into query requirements and analysis requirements.

[0034] As a specific example, the process of obtaining a pre-trained language model that incorporates a financial knowledge graph includes: Financial entities and their relationships are extracted from multi-source financial data to construct a financial knowledge graph. Financial entities include at least customers, accounts, products, risk events, and policies and regulations. The financial knowledge graph is injected into the pre-trained language model and trained to obtain a pre-trained language model that incorporates the financial knowledge graph.

[0035] Conventional pre-trained language models cannot meet the understanding needs of specialized technical fields. Therefore, it is necessary to address the pain points of traditional LLM in the financial field, such as "vague knowledge, weak reasoning, and insufficient compliance." By structuring relevant financial knowledge and integrating it into the language model, its semantic understanding and reasoning capabilities can be enhanced.

[0036] Specifically, it incorporates professional terminology and rules from the financial field, embedding banking business rules, such as interest rate calculation rules and risk assessment standards, into the model in the form of a knowledge graph. It also develops encryption processing specifications and compliance check operations for sensitive data in response to the bank's data security and compliance requirements.

[0037] Existing techniques for injecting knowledge graphs into pre-trained language models mainly fall into three categories: knowledge graph embedding (KGE) techniques (mapping entities and relations into vectors, then combining them with word vectors from the pre-trained language model via projection or attention mechanisms); inputting entities or relations from the knowledge graph as special tokens into the pre-trained language model, allowing it to learn knowledge representations during training; and designing joint training frameworks to enable mutual promotion between the knowledge graph and the language model, for example, updating outdated knowledge through dynamically editing knowledge graph embeddings (such as KGEditor). These existing techniques for fusing knowledge graphs with pre-trained language models can all achieve the desired effect of enhancing the semantic understanding capabilities of the pre-trained language model, as described in this application; therefore, these techniques will not be described in detail.

[0038] Furthermore, constructing a financial knowledge graph includes: Collect and preprocess multi-source financial data, clean up null and outlier values ​​and standardize field formats for structured data, and extract key information from unstructured data using financial NLP tools. A hybrid extraction scheme combining rules and machine learning is used to extract entities and relationships, generating a structured knowledge graph.

[0039] Specifically, this includes: defining graph patterns and service scenarios: clarifying the core business scenarios of graph services, such as "wide table query of deposit and loan data" and "wide table analysis of post-loan risk control data." Based on these scenarios, defining core entity types (such as customers, accounts, products, transactions, risk events, policies and regulations, etc.) and their key attributes. Simultaneously, defining high-frequency relationship types between entities, such as the "holding" relationship between customers and accounts, the "purchasing" relationship between accounts and products, the "triggering" relationship between transactions and risk events, and the "impact" relationship between policies and products.

[0040] Compliant Collection and Preprocessing of Multi-Source Financial Data: For structured data preprocessing, data in databases and wide tables is cleaned, null and outlier values ​​(such as negative transaction amounts) are handled, and field formats are standardized (e.g., dates are standardized to "YYYY-MM-DD"). For unstructured data preprocessing, for texts such as financial contracts, reports, and announcements, key information is extracted using financial NLP tools to prepare for subsequent steps, such as extracting "loan amount" and "guarantor" from contracts.

[0041] Hybrid Entity and Relation Extraction: For entity extraction, a domain-adaptive NER model based on BERT is used, combined with a financial terminology dictionary (such as "LPR interest rate" and "letter of credit") for entity identification and classification to improve accuracy. For relation extraction, a hybrid approach of "rules + machine learning" is adopted to balance accuracy and generalization.

[0042] For the rule layer of relation extraction, explicit relations are quickly and accurately identified based on rules such as dependency parsing (e.g., extracting the "apply" relation from "Zhang San applied for a mortgage"). For the model layer of relation extraction, a document-level relation extraction (DocRE) model is used to mine implicit relationships across sentences and texts from the global context (e.g., identifying the association risk implied by "The legal representative of Company A and the borrower of Company B share a mobile phone number").

[0043] As a specific example, generating corresponding data analysis code using a code generation model includes: The code generation model is a sequence-to-sequence model, which includes an encoder that encodes the analysis requirements description into a fixed-length vector representation, and a decoder that generates the corresponding code sequence based on the vector representation.

[0044] Specifically, the encoder encodes the analysis requirement description into a fixed-length vector representation, which fully contains the semantic information of the description; the decoder then generates the corresponding Python code sequence based on this vector representation. During the generation process, an attention mechanism is used to allow the decoder to dynamically focus on the part of the encoder output that is most relevant to the current generation step. For complex financial data analysis requirements, the model supports multiple rounds of generation and code snippet combination, and can handle the complete data analysis process, including data reading, data cleaning, feature engineering, statistical analysis, machine learning modeling, and result visualization.

[0045] When applied to broad-table data analysis scenarios in the financial sector, the method of this invention is specifically optimized for the unique business needs of banks, including adapting to typical scenarios such as broad-table queries of deposit and loan data and broad-table analysis of post-loan risk control data. In natural language query processing, special consideration is given to the expression habits of professional terminology commonly used by bank users and their fuzzy query needs, such as understanding specific contexts like "high-net-worth clients," "potentially risky accounts," and "fund flow tracking." In the data analysis code generation stage, banking business rules and compliance requirements are embedded to ensure that the generated statistical analysis, risk assessment, and regulatory reports comply with financial industry standards and regulations.

[0046] As an optional embodiment, such as Figure 2 As shown, after generating the corresponding data analysis code using the code generation model, a verification step is also set up: The generated data analysis code is validated, including syntax checking, functional logic verification, and data type compliance checks. If the validation fails, error messages are sent back to regenerate the data analysis code until the validation passes.

[0047] The generated data analysis code undergoes multi-level verification through a code verification module. First, a Python static code analysis tool is integrated for syntax checking and style verification to ensure the code conforms to programming standards. Then, functional test cases are set to verify the correctness of the code logic, including normal scenarios, abnormal scenarios, and boundary condition tests. At the same time, input and output data type verification and numerical range compliance checks are implemented to ensure that data processing complies with financial business rules. The code verification module also records a complete execution log, including input parameters, intermediate results, and output results. When verification fails, specific error information and suggestions are fed back to the code generation model for iterative optimization, forming a closed-loop code quality assurance mechanism.

[0048] Boundary condition testing includes, but is not limited to: testing processing capabilities with empty dataset input, verifying execution efficiency under maximum data load, checking computational accuracy with minimum data volume, simulating fault tolerance mechanisms under extreme numerical input, and evaluating system stability under concurrent access; specifically designed for financial business characteristics, financial-specific boundary tests are set up, such as compliance handling of special scenarios like zero-amount transactions, negative balances, and ultra-large-scale fund flows, to ensure that the generated code maintains stable execution performance and accurate result output under various edge conditions.

[0049] The purpose of setting up a code verification module in this invention to verify the data analysis code is to: determine whether the code can be executed correctly to generate results; if an error occurs during execution, the error information is returned to the verifier to correct the code until it can be executed correctly; determine whether the generated data results meet the requirements; if the data results are incorrect or the interface display is not intuitive, feedback is provided to adjust the data results and display format.

[0050] A wide-table data analysis system based on semantic association includes: The indicator library construction module builds and stores a preset indicator library. The natural language processing module receives and understands the user's natural language query requirements and generates corresponding semantic representations. The knowledge graph unit, set up in the natural language processing module, transforms diverse financial data into a structured knowledge graph; The semantic parsing module decomposes semantic expressions into logical expressions that include query and analysis requirements; the semantic parsing module includes: The preliminary parsing unit performs preliminary analysis of semantic expressions based on combinatorial category grammar, identifying key financial concepts and sentence structures; The deep parsing unit captures the dependencies between semantic units based on the preliminary parsing results and generates structured logical expressions; The query execution module generates and executes query statements based on the logical expression of the query requirements to obtain wide table data; The analysis and execution module generates and executes data analysis code based on the logical expression of the analysis requirements, and obtains the analysis results.

[0051] The code verification module verifies and corrects the generated data analysis code.

[0052] The indicator library construction module is responsible for creating and maintaining the system's core dictionary resources. This module constructs the indicator library through the following steps: First, collect wide-table data for various business scenarios, including wide-table names, field names, data types, and descriptions of business meanings. For banking operations, this needs to cover wide-table information in core business areas such as deposits and loans, payment and settlement, and risk management. Second, establish a business terminology dictionary, collecting synonyms, abbreviations, and related terms for each business indicator. For example, for the indicator "customer balance," it's necessary to collect synonyms such as "account balance," "deposit balance," and "balance query." Finally, construct a semantic association network, using word vector technology to calculate semantic similarity between terms and establish relationships between them.

[0053] The natural language processing module is the core component of the system of this invention, responsible for understanding the natural language queries input by the user. This module employs a pre-trained language model combined with a financial knowledge graph, and its specific implementation is as follows: First, prepare the training data. Collect text data from the financial sector, including business documents, query logs, customer service dialogues, etc., to build a domain-specific training corpus. The corpus needs to cover various business scenarios and query expressions.

[0054] Secondly, a financial knowledge graph is constructed. Then, model pre-training is performed, based on pre-trained models such as BERT or GPT, using domain adaptation techniques on financial corpora. During the training process, information from the knowledge graph is introduced to inject structured knowledge into the language model.

[0055] Finally, the query understanding function is implemented. When a user enters a natural language query, the model first performs word segmentation and entity recognition, and then combines the context information to understand the query intent. For example, for the query "find high-net-worth clients whose transaction amount exceeds 100,000 yuan in the most recent month", the model needs to identify the time condition "most recent month", the amount condition "exceeds 100,000 yuan", and the client type "high-net-worth clients".

[0056] The semantic parsing module is responsible for converting semantic representations into structured logical expressions. This module employs a hierarchical parsing strategy and specifically includes two units: The initial parsing unit is implemented based on Combinatorial Category Grammar (CCG). CCG is a highly lexical formalism that effectively handles the combinatorial nature of natural language. This unit predefines a set of grammatical rules and professional terminology categories in the financial field, enabling rapid identification of key components in queries.

[0057] The deep parsing unit is implemented based on a neural network, employing an architecture that combines a long short-term memory network with an attention mechanism. It can capture long-range semantic dependencies and understand complex query logic. For example, for complex queries containing multiple conditions, the deep parsing unit can accurately identify the logical relationships (AND, OR, NOT, etc.) between the conditions.

[0058] The query execution module is responsible for generating query statements and executing data query operations. This module first matches the corresponding wide table and field information from the indicator library based on the query requirements in the logical expression. The matching process can be based on semantic similarity calculations, using metrics such as cosine similarity. The results of the matching query are then filled into the corresponding positions in the logical expression to form a complete SQL statement. Finally, the generated SQL statement is executed to retrieve the target data from the wide table. The analysis execution module is responsible for data analysis and result generation. This module employs a code generation model, automatically generating data analysis code based on the analysis requirements. The code generation model is based on a sequence-to-sequence architecture, and the generated analysis code includes steps such as data preprocessing, feature engineering, statistical analysis, and machine learning modeling. The code execution engine supports data analysis languages ​​such as Python and R, and can call various data analysis libraries and algorithms.

[0059] The above embodiments are further elaborations and descriptions of the present invention to facilitate understanding, and are not intended to limit the present invention in any way. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A wide-table data analysis method based on semantic association, characterized in that, include: It receives natural language query requests from users, uses a pre-trained language model combined with financial knowledge graphs for semantic understanding, and generates corresponding semantic representations. Semantic parsing is performed on the semantic representation, breaking it down into logical expressions that include query requirements and analysis requirements; Based on the logical expression of the query requirements, the corresponding wide table and field information are matched through the preset indicator library to generate an executable query statement and obtain the target data from the wide table. Based on the logical expression of the analysis requirements, the code generation model is used to generate corresponding data analysis code, which is then used to analyze the target data and generate analysis results.

2. The wide table data analysis method based on semantic association according to claim 1, characterized in that, The semantic parsing of the semantic expression, breaking it down into logical expressions that include query and analysis requirements, includes: Combinatorial category grammar is used to perform preliminary analysis of semantic expressions, identifying key financial concepts and sentence structures; The initial parsing results are input into a deep parsing unit based on a neural network to capture the dependencies between semantic units and generate structured logical expressions.

3. A wide table data analysis method based on semantic association according to claim 1 or 2, characterized in that, The process of generating corresponding data analysis code using a code generation model includes: The code generation model is a sequence-to-sequence model, which includes an encoder that encodes the analysis requirement description into a fixed-length vector representation, and a decoder that generates a corresponding code sequence based on the vector representation.

4. A wide table data analysis method based on semantic association according to claim 1 or 2, characterized in that, The process of obtaining the pre-trained language model combined with the financial knowledge graph includes: Financial entities and relationships between entities are extracted from multi-source financial data to construct a financial knowledge graph. The financial entities include at least customers, accounts, products, risk events, and policies and regulations. The financial knowledge graph is injected into a pre-trained language model and trained to obtain a pre-trained language model that incorporates the financial knowledge graph.

5. The wide table data analysis method based on semantic association according to claim 4, characterized in that, The construction of the financial knowledge graph includes: Collect and preprocess multi-source financial data, clean up null and outlier values ​​and standardize field formats for structured data, and extract key information from unstructured data using financial NLP tools. A hybrid extraction scheme combining rules and machine learning is used to extract entities and relationships, generating a structured knowledge graph.

6. A wide table data analysis method based on semantic association according to claim 1, 2, or 5, characterized in that, The preset indicator library includes at least the wide table name, the field names within the wide table, synonyms and abbreviations of the indicator names.

7. A wide table data analysis method based on semantic association according to claim 1, 2, or 5, characterized in that, After generating the corresponding data analysis code using the code generation model, a verification step is also set: The generated data analysis code is validated, including syntax checking, functional logic verification, and data type compliance checking. If the validation fails, an error message is sent back to regenerate the data analysis code until the validation passes.

8. A wide-table data analysis system based on semantic association, applicable to the wide-table data analysis method as described in any one of claims 1-7, characterized in that, include: The indicator library construction module builds and stores a preset indicator library. The natural language processing module receives and understands the user's natural language query requirements and generates corresponding semantic representations. The semantic parsing module breaks down semantic expressions into logical expressions that include query and analysis requirements; The query execution module generates and executes query statements based on the logical expression of the query requirements to obtain wide table data; The analysis and execution module generates and executes data analysis code based on the logical expression of the analysis requirements, and obtains the analysis results.

9. A wide-table data analysis system based on semantic association according to claim 8, characterized in that, Also includes: The knowledge graph unit, set up in the natural language processing module, transforms diverse financial data into a structured knowledge graph; The code verification module verifies and corrects the generated data analysis code.

10. A wide-table data analysis system based on semantic association according to claim 8 or 9, characterized in that, The semantic parsing module includes: The preliminary parsing unit performs preliminary analysis of semantic expressions based on combinatorial category grammar, identifying key financial concepts and sentence structures; The deep parsing unit captures the dependencies between semantic units based on the preliminary parsing results and generates structured logical expressions.

Images