A data analysis method, system and storage medium

Abstract

The present application discloses a data analysis method, system and storage medium for the field of natural language processing. The present application method includes: obtaining natural language input by a user; constructing a semantic data model that converts the natural language into a semantic query statement; combining the semantic data model with custom rules to construct a semantic parser; inputting the natural language into the semantic parser for parsing to obtain a semantic query statement; inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement; converting the target semantic query statement into a structured query statement through a semantic translator; inputting the structured query statement into the database for query to obtain a query result.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular to a data analysis method, system and storage medium. Background Art

[0002] With the rapid development of science and technology, user query needs are becoming increasingly diverse and complex. Users expect to obtain the required information quickly and accurately through the query system. However, traditional query methods often rely on strict grammatical rules and keyword matching, which to a certain extent limits the user's query freedom. When users enter query information in the form of natural language, if there are slight differences in grammar or logic, the query system may not be able to correctly understand the user's intention, which may lead to query failure or inaccurate return results.

[0003] To solve this problem, existing technologies have introduced natural language processing technology, which can convert the natural language input by users into a format that the query system can understand, thereby effectively reducing the query failure rate and improving the accuracy of the returned results.

[0004] However, natural language processing technology still has deficiencies in semantic analysis and understanding. In particular, when dealing with complex or ambiguous expressions, existing natural language processing technology often has difficulty accurately capturing the true intention of user input. This not only affects the accuracy of query results, but also limits the application of query systems and user experience. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data analysis method, system and storage medium.

[0006] The technical solution provided in this application is described below:

[0007] The first aspect of the present application provides a data analysis method, the method comprising:

[0008] Get the natural language input by the user;

[0009] Build the initial data model;

[0010] Perform word segmentation on the natural language according to a preconfigured industry knowledge base and a WordPiece word segmenter to construct a prompt;

[0011] Encode the prompt through the pre-trained model to obtain the encoding vector of each token;

[0012] The encoding vector of the first classification tag token output by the pre-trained model is used as the entire sentence representation;

[0013] Processing all the encoding vectors according to the sentence representation to obtain a local semantic representation;

[0014] By identifying entities and attributes in the natural language, and mapping the entities and attributes to Schema elements of a database, a mapping relationship is obtained;

[0015] Constructing a semantic data model according to the mapping relationship and the local semantic representation;

[0016] Combining the semantic data model with custom rules to build a semantic parser;

[0017] Inputting the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0018] Inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement;

[0019] Converting the target semantic query statement into a structured query statement by a semantic translator;

[0020] Inputting the structured query statement into the database for query to obtain query results; constructing a semantic data model according to the mapping relationship and the local semantic representation includes:

[0021] Initialize the mapping relationship based on the context information of the currently input natural language query

[0022] Adjusting the weight of the Schema element of the database according to the context information;

[0023] Constructing an initial semantic graph, wherein the initial semantic graph includes entity nodes, relationship nodes and operation nodes, entity relationship edges, conditional relationship edges and operation relationship edges;

[0024] For each node, combining the local semantic representation, metadata information of the database, and historical query behavior of the user to generate a comprehensive node feature representation of each node;

[0025] Initializing features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation;

[0026] Embed the comprehensive node feature representation and the comprehensive edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on a sink algorithm that implements a connector to obtain an input semantic graph;

[0027] The input semantic graph is input into a pre-built initial graph neural network for training, the information of neighbor nodes is aggregated through the graph convolution operation of each layer in the initial graph neural network, and the status of each node is updated in combination with the characteristics of each node;

[0028] The initial graph neural network is optimized using the Adam optimizer to obtain a converged semantic data model.

[0029] Optionally, the prompt is encoded using a pre-trained model to obtain an encoding vector for each token, including:

[0030] Convert the prompt into the input format of the pre-trained model to obtain input_ids and attention_mask;

[0031] The input_ids and the attention_mask are input into the pre-trained model to obtain the encoding vector of each token.

[0032] Optionally, identifying entities and attributes in the natural language and mapping the entities and attributes to Schema elements of a database to obtain a mapping relationship includes:

[0033] Extracting key entities in the natural language;

[0034] Query database schema information to obtain the Schema element of the database;

[0035] The key entity is matched with the Schema element to obtain a mapping relationship.

[0036] Optionally, inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement includes:

[0037] Construct initialization semantic modifier;

[0038] Determining whether the semantic query statement complies with a preconfigured correction rule;

[0039] If not, input the semantic query statement into a preconfigured machine learning model to obtain correction suggestions;

[0040] Determining whether there are more than three query conditions in the semantic query statement;

[0041] If yes, decomposing the semantic query statement to obtain a single query condition statement;

[0042] Performing semantic correction on the single query condition statement according to the correction suggestion to obtain a corrected query statement;

[0043] Evaluate the query statement and obtain evaluation results;

[0044] Improving the preconfigured machine learning model and the correction rule according to the evaluation effect to obtain a target semantic corrector;

[0045] The semantic query statement is input into the target semantic corrector for correction to obtain a target semantic query statement.

[0046] Optionally, after converting the target semantic query statement into a structured query statement by a semantic translator, the method further includes:

[0047] Determine whether there is a statement error in the structured query statement according to the rule engine;

[0048] If yes, input the structured query statement into a preconfigured deep learning model to obtain the statement error type;

[0049] Determine whether the statement error type is a field name error;

[0050] If not, a prompt selection is issued to the user to obtain a selection result;

[0051] The structured query statement is adjusted according to the selection result to obtain a target structured query statement.

[0052] Optionally, inputting the structured query statement into the database for query to obtain a query result includes:

[0053] Inputting the target structured query statement into the database for query to obtain query data;

[0054] Selecting a data chart according to the data type of the query data;

[0055] The query data and the data chart are visually displayed to obtain query results.

[0056] Optionally, selecting a data chart according to the data type of the query data includes:

[0057] Extracting statistical features of the query data;

[0058] Inputting the statistical features into a preconfigured recognition model to obtain a distribution type;

[0059] A data chart is generated according to the distribution type.

[0060] A second aspect of the present application provides a data analysis system, the system comprising:

[0061] An acquisition unit, used for acquiring natural language input by a user;

[0062] A first construction unit is used to construct an initial data model;

[0063] The second construction unit is used to perform word segmentation processing on the natural language according to a preconfigured industry knowledge base and a WordPiece word segmenter to construct a prompt;

[0064] An encoding unit, used to encode the prompt through a pre-trained model to obtain an encoding vector for each token;

[0065] A first processing unit, configured to use the encoding vector of the first classification tag token output by the pre-trained model as the entire sentence representation;

[0066] A second processing unit, configured to process all the encoding vectors according to the sentence representation to obtain a local semantic representation;

[0067] A third processing unit is used to identify entities and attributes in the natural language and map the entities and attributes to Schema elements of a database to obtain a mapping relationship;

[0068] A third construction unit, configured to construct a semantic data model according to the mapping relationship and the local semantic representation;

[0069] A fourth construction unit, configured to construct a semantic parser by combining the semantic data model with a custom rule;

[0070] A fourth processing unit, configured to input the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0071] A fifth processing unit, configured to input the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement;

[0072] A sixth processing unit, configured to convert the target semantic query statement into a structured query statement through a semantic translator;

[0073] A query unit, used for inputting the structured query statement into the database for querying to obtain a query result;

[0074] The third building block is specifically used for:

[0075] Initialize the mapping relationship based on the context information of the currently input natural language query

[0076] Adjusting the weight of the Schema element of the database according to the context information;

[0077] Constructing an initial semantic graph, wherein the initial semantic graph includes entity nodes, relationship nodes and operation nodes, entity relationship edges, conditional relationship edges and operation relationship edges;

[0078] For each node, combining the local semantic representation, metadata information of the database, and historical query behavior of the user to generate a comprehensive node feature representation of each node;

[0079] Initializing features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation;

[0080] Embed the comprehensive node feature representation and the comprehensive edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on a sink algorithm that implements a connector to obtain an input semantic graph;

[0081] The input semantic graph is input into a pre-built initial graph neural network for training, the information of neighbor nodes is aggregated through the graph convolution operation of each layer in the initial graph neural network, and the status of each node is updated in combination with the characteristics of each node;

[0082] The initial graph neural network is optimized using the Adam optimizer to obtain a converged semantic data model.

[0083] A third aspect of the present application provides a data analysis system, the system comprising:

[0084] Processor, memory, input-output unit, and bus;

[0085] The processor is connected to the memory, the input and output unit, and the bus;

[0086] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0087] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0088] It can be seen from the above technical solutions that this application has the following advantages:

[0089] 1. This application uses a pre-configured industry knowledge base and WordPiece word segmenter to segment natural language to construct a prompt, and uses a pre-trained model to encode the prompt, which can accurately capture the semantic information in natural language and improve the natural language analysis and processing capabilities. Then, the first classification tag token obtained after encoding is used as the entire sentence representation, and all encoding vectors are processed according to the sentence representation to obtain a local semantic representation. The local semantic representation can reflect the complex semantic structure in natural language, help to better understand the user input intention, and provide a basis for subsequent semantic conversion. Further, by mapping entities and attributes in natural language to database Schema elements to obtain a mapping relationship, this step establishes a bridge between natural language and structured data, so that unstructured natural language can be effectively organized and utilized. Finally, a semantic data model is constructed through mapping relationships and local semantic representations, and then the semantic data model is combined with custom rules to construct a semantic parser. The constructed semantic parser improves the accuracy and efficiency of converting natural language into semantic query statements, provides a basis for the subsequent conversion of semantic query statements into structured query statements, and also enhances the system's semantic understanding and analysis capabilities;

[0090] 2. By inputting the semantic query statement into the semantic corrector for correction, the semantic query statement is further optimized, the ambiguity and errors in the semantic query statement are eliminated, the accuracy of the subsequent query results is improved, and the natural language analysis capability is further improved;

[0091] 3. Translate the target semantic query statement through the semantic translator. This step converts the semantic query statement into a structured query statement that can be recognized by the machine, which is convenient for database understanding and execution, improves query efficiency and accuracy, and provides users with a better query experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0093] Figure 1 A schematic diagram of an embodiment of the data analysis method provided in this application;

[0094] Figure 2 A flowchart of another embodiment of the data analysis method provided in the present application;

[0095] Figure 3 A flowchart of another embodiment of the data analysis method provided in the present application;

[0096] Figure 4 A flowchart of another embodiment of the data analysis method provided in the present application;

[0097] Figure 5 A flowchart of another embodiment of the data analysis method provided in the present application;

[0098] Figure 6 A flowchart of another embodiment of the data analysis method provided in the present application;

[0099] Figure 7 A schematic diagram of an embodiment of a data analysis system provided by the present application;

[0100] Figure 8 A flowchart of another embodiment of the data analysis system provided in the present application. DETAILED DESCRIPTION

[0101] It should be noted that the data analysis method provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as an example for explanation.

[0102] See also Figure 1 The present application first provides an embodiment of a data analysis method, which includes:

[0103] S101, obtaining natural language input by a user;

[0104] Natural language refers to the language used by humans in daily life, including spoken and written language. In this embodiment, natural language refers to the natural question and answer sentences entered by the user into the search box.

[0105] In this embodiment, before obtaining the natural language input by the user, a user input interface needs to be provided, and then the user enters the question and answer requirements on the interface through a keyboard, voice or other input methods. These input question and answer requirements are captured by the system in the form of data, which helps to generate query results that meet the user's expectations in the future.

[0106] S102, constructing an initial data model;

[0107] In this embodiment, before constructing the initial data model, the data source type must first be configured, then the data source to be queried must be identified, and the corresponding data source configuration must be performed. After the data source configuration is completed, the data model can be constructed, and the constructed data model needs to construct its dimensions, metrics, indicators, and associated keys.

[0108] In this embodiment, constructing the initial data model provides a basis for the subsequent acquisition of the semantic data model. The dimensions, metrics, indicators, and associated keys of the data model are constructed to ensure the accuracy and relevance of the data.

[0109] S103, performing word segmentation processing on the natural language according to a preconfigured industry knowledge base and a WordPiece word segmenter to construct a prompt;

[0110] An industry knowledge base is a customized resource library that integrates expertise, terminology, and corresponding data structures in a specific field. It is designed to provide convenient, accurate, and comprehensive knowledge support for models or related systems. The WordPiece tokenizer is an algorithm for text processing that is mainly used to decompose words into subwords or tokens. A prompt is a designed text input, usually a request or question to the model.

[0111] In this embodiment, before the natural language is segmented, it is necessary to integrate domain expertise, terminology, and data structures to configure an industry knowledge base. The WordPiece word segmenter is then used to initialize the vocabulary based on the industry knowledge base, and the probability of each word in the natural language is calculated to obtain the initial likelihood value of the natural language. Next, the WordPiece word segmenter selects adjacent subwords with the largest mutual information value to merge, form new subwords, and update the vocabulary. After each merger, the likelihood value of the sentence is recalculated to evaluate the impact of the merger on the overall probability of the sentence until the predetermined vocabulary size is reached, and finally a prompt of appropriate length is obtained.

[0112] In practical applications, suppose a sentence consisting of n words represents the i-th word, and the language model likelihood value of that sentence T can be expressed as the product of all word probabilities. Then, the adjacent words a and b are merged into a new word c, and then the likelihood values ​​of the sentences before and after the merger are calculated. Then, the WordPiece word segmenter is used to select the two adjacent subwords with the largest mutual information value for merging. After merging the two words, replace them with a new subword and update the vocabulary. This process is iterated until the predetermined vocabulary size is reached. For example, for the query "What is the sales volume of TV sets?" After the above preprocessing, the pre-trained model input sequence obtained is: [CLS] "TV sets", "of", "sales volume", "is", "how much"?

[0113] In this embodiment, by using the preconfigured industry knowledge base and the WordPiece word segmenter to perform word segmentation processing on natural language, the words and terms in the text can be accurately identified and segmented, and then by reconstructing these words and terms, a prompt of the appropriate length required by the pre-trained model can be obtained, which provides assistance for the subsequent encoding of the prompt using the pre-trained model.

[0114] S104, encoding the prompt through a pre-trained model to obtain an encoding vector for each token;

[0115] Pre-trained models refer to models trained by deep learning algorithms on large data sets. These models learn a large number of common features and knowledge representations during the training process and can be used for initialization solutions for various related tasks. In this embodiment, three pre-trained models, BERT, T5 and CodeBERT, are used to train data models. In terms of the selection of pre-trained models, three strategies are formulated, namely task specificity, model size and domain adaptability. Task specificity refers to the selection of models related to SQL generation tasks; model size refers to the balance between the size of the model and the available computing resources. Large models may be better than small models in terms of accuracy, but they will also increase computing overhead; domain adaptability refers to the selection of models pre-trained in related fields (such as finance, medical care, and e-commerce) to better understand domain-specific terms and relationships. Token refers to the smallest unit or element into which the prompt is divided. It can be a word, subword (such as a root, affix), character or other text fragment.

[0116] In this embodiment, before prompt is encoded, prompt needs to be converted into the input format of the pre-trained model, that is, converted into input_ids and attention_mask required by the pre-trained model, where input_ids is the index of each token, indicating the representation in the vocabulary of the pre-trained model, and attention_mask indicates which positions are valid. Then, input_ids and attention_mask are input into the pre-trained model for processing to obtain the encoded token vector representation, that is, the encoding vector of the token.

[0117] S105, using the encoding vector of the first classification mark token output by the pre-training model as the entire sentence representation;

[0118] The classification token is [CLS]token, which is a special token mainly used for classification tasks. It is placed at the beginning of the input sequence to indicate the category or overall semantics of the entire input sequence. Sentence representation refers to the meaning of the entire sentence.

[0119] In this embodiment, the encoding vector of the first classification token output by the pre-training model is used as the representation of the entire sentence. The encoding vector of the classification token captures the overall semantic information of the sentence and provides a basis for subsequent acquisition of local semantic representation.

[0120] S106, processing all the encoding vectors according to the sentence representation to obtain a local semantic representation;

[0121] In this embodiment, based on the obtained sentence representation, the encoding vectors of all tokens in the sentence are processed to extract local semantic information in the sentence. This process can obtain a detailed local semantic representation by analyzing the encoding vector of each token, which provides a basis for the subsequent construction of a semantic data model, wherein the local semantic representation is used for specific field matching and semantic SQL generation.

[0122] S107, identifying entities and attributes in the natural language, and mapping the entities and attributes to Schema elements of a database to obtain a mapping relationship;

[0123] In this embodiment, entities and attributes in natural language refer to specific people, things and objects. For example, the query statement entered by the user is "show all orders with sales exceeding 1000", where "sales" is the entity and attribute of this query statement. The Schema elements of the database mainly include tables, constraints, views, stored procedures, and data types, etc. These elements together constitute the structure and function of the database, define the storage method of data, the relationship between data, and how to operate these data. A mapping relationship refers to the relationship between the elements of two data that correspond to each other.

[0124] In this embodiment, before obtaining the mapping relationship, it is necessary to first obtain the natural language, then identify the entities and attributes in the natural language, and then extract the key entities of the natural language based on the identified entities and attributes. Further, the database schema is parsed to obtain the schema information of the database, list all Schema elements, and finally match the key entities in the natural language with the Schema elements in the database to obtain the mapping relationship.

[0125] In actual applications, the user input "Show all orders with sales exceeding 1000" is obtained, and then the key entities such as "sales" and "orders" are extracted by identifying the entities and attributes in "Show all orders with sales exceeding 1000". Then, the database is parsed to obtain all related tables and columns. For example, the table orders has the columns: id, sales, customer_id, and order_date. Further, "sales" is matched to the sales column, and "orders" is matched to the orders table. The relationship that matches the key entities to the related tables or columns is the mapping relationship.

[0126] This embodiment identifies entities and attributes in natural language and maps them to the Schema elements of the database, thereby establishing a mapping relationship between natural language and database structure, so that the system can quickly understand the user's query information, reduce errors in manual intervention, and provide a basis for later generation of a model that converts natural language into semantic query statements.

[0127] S108, constructing a semantic data model according to the mapping relationship and the local semantic representation;

[0128] In this embodiment, to construct a semantic data model, it is necessary to obtain the local semantic representation and the mapping relationship between the natural language and the database Schema elements, and then construct a large model that converts the natural language into semantic query statements based on the mapping relationship and the local semantic representation, namely the semantic data model.

[0129] In this embodiment, a semantic data model is constructed to realize the conversion of natural language into semantic query statements. When using the semantic data model, the model can accurately extract key entities in natural language, and then generate corresponding semantic query statements using the internally set mapping relationship and local semantic representation, which greatly improves the efficiency and accuracy of subsequent queries and also provides a basis for the subsequent construction of a semantic parser.

[0130] Specifically, a possible implementation of this step includes:

[0131] Step 1: Initialize the mapping relationship according to the context information of the natural language query currently input;

[0132] In this implementation, the input natural language query is first parsed through the natural language processing (NLP) module to extract key information, such as the target entity of the query, query conditions, attributes, operation type (such as SELECT, WHERE, etc.) and its context.

[0133] For example: "Find people older than 30 years old" → Extract "age" as an attribute, "30 years old" as a condition, and "person" as an entity.

[0134] Analyze the context of the query to determine the context and intent of the query. For example, if the query involves complex aggregation or conditional operations (such as AVG, SUM), the weights of different parts need to be adjusted based on the context.

[0135] Context information may include user historical queries, query frequency, frequency of use of query fields, etc.

[0136] Based on the extracted query elements and context information, natural language elements (such as entities, attributes, and operations) are mapped to Schema elements (such as tables and columns) and relationships in the database, and the mapping relationship between each component in the query is initialized.

[0137] Step 2: adjusting the weight of the Schema element of the database according to the context information;

[0138] Dynamically adjust the weights of Schema elements (such as tables and columns) in the database based on the context of the input query. For example, if a user frequently queries a table or column, the importance of that table or column in the current query is increased. Assign different initial weights to each entity, column, and operation, and weight them based on the query context. For commonly used tables / columns, higher weights may be assigned so that the model prioritizes these high-weight elements when generating queries. Analyze the user's historical query behavior to determine which Schema elements are most relevant, and adjust the weights based on the user's query habits. This can help the model more accurately generate query structures that users may be interested in.

[0139] Step 3: construct an initial semantic graph, wherein the initial semantic graph includes entity nodes, relationship nodes, operation nodes, entity relationship edges, conditional relationship edges, and operation relationship edges;

[0140] This step is to construct nodes and edges. For different nodes and edges, there are the following:

[0141] Entity node: maps to database tables and columns, such as Customer, age.

[0142] Relationship node: represents the relationship between tables, such as Customer.id = Order.customer_id.

[0143] Operation node: indicates the SQL operation type, such as SELECT, WHERE, and JOIN.

[0144] Side Build:

[0145] Entity relationship edge: For example, Customer.id = Order.customer_id.

[0146] Conditional relationship edge: for example, age>30.

[0147] Operation relationship edge: for example, the relationship between SELECT and the target field, and the relationship between WHERE and the condition.

[0148] Based on the above nodes and edges, an initial semantic graph is constructed. In this graph, nodes represent the core elements of the query (tables, columns, conditions, etc.), and edges represent the relationships and operations between these elements.

[0149] Step 4: for each node, combining the local semantic representation, metadata information of the database, and historical query behavior of the user, generate a comprehensive node feature representation of each node;

[0150] Generate local semantic representations (such as word vectors, embedding representations, etc.) for each node (such as table name, column name, condition, etc.) based on the context of the natural language query.

[0151] For example, you can generate word vector representations for the table name Customer and the column name age. Combine the database metadata (such as field type, table relationship, index, etc.) to generate additional feature information for the node. For example, the data type of the age column is an integer, and the "integer type" tag can be added to its feature vector. Combine the user's query history to generate personalized feature representations. If the user has frequently queried the age field in the past, the weight will be higher, affecting the generation of node feature representations. By combining local semantics, metadata, and historical behavior, a comprehensive feature representation of each node is generated. These feature representations will be used as input to the graph neural network.

[0152] Step 5: Initialize the features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation;

[0153] When initializing edge features, it is necessary to consider the type of relationship it represents. For example, the features of the edge of a JOIN relationship and the edge of a WHERE condition may be different. A JOIN edge may contain a join condition, while a WHERE edge may contain a filter condition. The features of the edge can also be adjusted based on the query context. For example, if the query involves joining multiple tables, the weight of the edge can reflect the complexity of the connection. Combining the features of the node, the type of relationship, and the query context, a feature representation is generated for each edge. These edge feature representations will be propagated in subsequent graph convolution operations.

[0154] Step 6: embed the comprehensive node feature representation and the comprehensive edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on the sink algorithm that implements the connector to obtain an input semantic graph;

[0155] The comprehensive feature representation of each node and edge is embedded into the initial semantic graph to form the input semantic graph. The feature representation of each node and edge will be used as its input in the graph neural network. Ensure that the structure of the graph can effectively capture the semantics and relationships of the query.

[0156] In step 6, the embedding model TransE can be used to embed the semantic graph. Specifically:

[0157] Map entities h, t and relations r into low-dimensional vectors ;

[0158] Define the optimization goal: The optimization goal in the model is: T;

[0159] Train the embedding model by optimizing the function to obtain the embedding vector;

[0160] Based on the similarity rules of the embedding space, the semantic graph content is expanded:

[0161] Logical rules based on the embedding space, such as path reasoning, complement the relationships that are not explicitly labeled.

[0162] Embed and map the newly added data in real time, and add new entities and relationships to the semantic graph. Specifically:

[0163] Incrementally train the embedding model to avoid reconstructing the entire embedding space;

[0164] Update the semantic graph structure so that it can dynamically reflect new semantics;

[0165] The stream-batch integrated technology framework can be used to implement the sink algorithm of the connector to write the newly added mapping relationship data into the semantic graph in real time or offline to achieve the effect of dynamic update.

[0166] Step 7: Input the input semantic graph into a pre-built initial graph neural network for training, aggregate the information of neighbor nodes through the graph convolution operation of each layer in the initial graph neural network, and update the status of each node in combination with the characteristics of each node;

[0167] Each layer of the graph neural network collects information from neighboring nodes through graph convolution operations and aggregates features between nodes. For example, in a JOIN operation, the graph convolution operation aggregates node features related to the Customer and Order tables to update the node status and understand the relationship between them. Combined with the aggregated neighbor information, the status of each node is updated. Through multiple layers of graph convolution, the model can gradually capture more complex semantic relationships.

[0168] Step 8: Use the Adam optimizer to optimize the initial graph neural network to obtain a converged semantic data model.

[0169] In this implementation, the Adam optimizer is used to optimize the parameters of the graph neural network and minimize the loss function. The goal of the optimization is to enable the graph neural network to accurately map the input natural language query to the SQL query statement. The mean square error can be used for optimization. Through multiple rounds of training, the weights of the network are updated until the model converges, that is, the difference between the output SQL query statement and the real query is minimized.

[0170] S109, constructing a semantic parser by combining the semantic data model with custom rules;

[0171] The semantic parser is an important tool in natural language processing technology. In this embodiment, its main function is to convert natural language text into semantic query statements so that the system can understand and execute human instructions. Custom rules refer to a series of rules defined manually according to specific needs or business logic. These rules are used to guide the semantic parser on how to understand and process natural language input, and how to convert it into corresponding semantic representations or query statements. Custom rules may include, but are not limited to: defining how to identify key entities and attributes in natural language and map them to Schema elements of the database; specifying the grammatical structure and semantic requirements of natural language input to ensure the validity and accuracy of the input; defining how to convert query conditions in natural language into filter conditions in database query statements.

[0172] In this embodiment, to build a semantic parser, it is first necessary to obtain a trained semantic data model, then build some custom rules, and then build a semantic parser based on the semantic data model and the custom rules to convert the user's natural language into a standardized semantic query statement. By building a semantic parser, it helps to convert the user's natural language into a standardized semantic query statement in the future, and provides a basis for obtaining accurate query results in the later stage.

[0173] Specifically, a possible implementation of this step includes:

[0174] Step a: constructing a standardized query template framework, wherein the standardized query template framework includes selection operations, filtering conditions, sorting and grouping operations;

[0175] Define a select operation template to specify the target entities or attributes to be returned in the query. The select operation is the first part of the SQL query.

[0176] Example: For the query "find the customer's name", the template framework converts it to: SELECT name FROMCustomer

[0177] Filter condition (WHERE):

[0178] Defines a filter condition template to specify constraints in a query. Filter conditions usually correspond to the WHERE clause and are used to specify which records meet the query conditions.

[0179] Example: For the query "Find customers older than 30 years old", the template framework converts it to: SELECT name FROM Customer WHERE age>30

[0180] Sorting and grouping (ORDER BY, GROUP BY):

[0181] Define sorting rules to map the sorting conditions in the query to the ORDER BY statement in SQL.

[0182] Example: Query "List of customers sorted by age" → SELECT name, age FROM CustomerORDER BY age

[0183] Define grouping rules and map the grouping conditions in the query to the GROUP BY statement in SQL.

[0184] Example: Query "number of customers grouped by city" → SELECT city, COUNT(*) FROM CustomerGROUP BY city

[0185] This standardized framework ensures that natural language queries can be converted into structured, standardized SQL queries.

[0186] Step b: construct entity mapping rules for mapping entities in natural language queries to tables and columns in the semantic data model;

[0187] Use natural language processing (NLP) techniques to identify entities in queries. Entities are usually nouns or noun phrases that represent specific objects in the database (such as customers, products, orders, etc.).

[0188] Example: The query "find the name of a customer", Customer is an entity.

[0189] For each identified entity, mapping rules are defined to convert it into tables and columns in the database.

[0190] For example, "customer" may be mapped to the Customer table in the database, and "name" may be mapped to the name column. It is also necessary to handle possible ambiguity and ensure that the same entity is mapped correctly in different contexts.

[0191] For example, a Customer may have multiple attributes such as name and address, so the specific fields of the entity need to be determined based on the context. Through entity mapping rules, you can ensure that the entities in the query can be correctly mapped to the database tables and columns in the semantic data model.

[0192] Step c: construct condition mapping rules to map the conditions in the natural language query to standard operators in the database;

[0193] Based on the query target, determine the entities or attributes that need to be selected. For example, the query "find customer name" will generate SELECT name FROM Customer. Generate the WHERE clause of the query through the condition mapping rules. For example, the query "age greater than 30" will be mapped to WHERE age>30. Use custom sorting and grouping rules to handle ORDER BY and GROUP BY in the query.

[0194] For example, the query "sort by age" would be mapped as ORDER BY age.

[0195] Step d: Construct sorting and grouping rules to process sorting and grouping operations in natural language queries, mapped to ORDER BY and GROUP BY in SQL statements;

[0196] For queries involving multiple tables, join rules are used to generate JOIN operations. For example, in the query "find the relationship between customers and orders", a custom rule is used to determine the use of JOIN to connect the Customer and Order tables, and the condition may be Customer.id = Order.customer_id.

[0197] Step e: construct subquery rules to process complex queries containing subqueries and generate corresponding query commands;

[0198] For queries containing subqueries, nested query commands are generated using subquery rules. Nested SQL subqueries are generated by decomposing the structured part of the query.

[0199] Step f: Construct a join rule to process the join operation between multiple tables and generate the corresponding SQL join conditions;

[0200] By applying all rules to specific natural language queries, the final SQL query statement can be integrated. The output of custom rules (such as entities, conditions, sorting, etc.) are combined into a SQL query structure that conforms to the semantic data model.

[0201] For example, for the natural language query "find all customers older than 30 and sort by age in ascending order", the final SQL query can be:

[0202] SELECT name FROM Customer WHERE age>30 ORDER BY age ASC

[0203] Step g: construct a filling relationship that embeds the output of the custom rule into the standardized query template framework to obtain a semantic parser.

[0204] Embed the output of the custom rule into the standardized query template framework, fill in the placeholders, and generate a complete SQL query. Specifically, fill the output of the custom rule (such as table name, column name, condition, etc.) into the placeholders of the standardized query template framework. Assume that the output of the custom rule is as follows:

[0205] columns → name, age

[0206] table → Customer

[0207] conditions → age>30

[0208] group_by → empty (no grouping)

[0209] order_by → ORDER BY age ASC

[0210] SQL query after filling:

[0211] SELECT name, age

[0212] FROM Customer

[0213] WHERE age>30

[0214] ORDER BY age ASC;

[0215] You also need to create a mapping mechanism to populate relationships for different types of placeholders (such as {columns}, {table}, {conditions}, etc.)

[0216] The core part of the semantic parser is query parsing and filling based on defined rules and templates. Flexible query parsing is achieved by designing appropriate data structures (such as rule lists, mapping tables, template engines, etc.).

[0217] The parser first maps natural language queries to corresponding rule outputs (e.g., entities, conditions, sorting, etc.), and then populates these outputs into the standardized query template framework.

[0218] Generate a complete SQL query statement based on the filled query template.

[0219] S110, inputting the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0220] In this embodiment, before obtaining a semantic query statement, it is necessary to obtain the natural language input by the user, and then input the natural language into the semantic parser. The semantic parser calls the semantic data model according to the internally set custom rules, and inputs the natural language into the semantic data model, and then the semantic data model processes the natural language to obtain a semantic query statement.

[0221] S111, inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement;

[0222] The semantic corrector is mainly used to correct semantic errors in semantic query statements to ensure that the semantic query statements are expressed more accurately and naturally. The semantic corrector deeply analyzes the semantic meaning of semantic query statements, discovers and corrects semantic deviations, and makes the expression of semantic query statements more in line with expectations.

[0223] In this embodiment, before correcting the semantic query statement, an initial semantic corrector must first be constructed. Then determine whether the statement complies with the preset correction rules. If it does not comply with the correction rules, it is input into the pre-trained machine learning model to obtain correction suggestions. If it complies with the correction rules, the target semantic query statement is directly output. Next, check whether the query conditions in the semantic query statement exceed three. If more than three, decompose the structure of the semantic query statement to obtain a query statement with a single query condition. Subsequently, semantic correction is performed on the query statement with each single query condition according to the correction suggestions. If no more than three, semantic correction is performed directly to obtain the corrected query statement. Further, the corrected query statement is evaluated, and the machine learning model and the correction rules are optimized and improved according to the evaluation results until the target semantic corrector is constructed. Finally, the original semantic query statement is input into this target semantic corrector to obtain the optimized target semantic query statement.

[0224] In this embodiment, the semantic query statement is input into the semantic corrector for semantic correction, which can significantly improve the accuracy of the query statement, ensure that the query result is more in line with user expectations, and reduce resource waste caused by erroneous queries.

[0225] S112, converting the target semantic query statement into a structured query statement by a semantic translator;

[0226] The semantic translator is a tool that can convert semantic query statements into structured query statements. Based on natural language processing and semantic understanding technology, it can deeply parse the semantic information in the query statement and accurately convert it into a structured query statement that can be understood and executed by the database or other systems. Among them, the structured query statement is a programming language specifically used to interact with the database. It provides a set of tools for operating the database, including functions such as querying, inserting, updating and deleting data.

[0227] In this embodiment, before obtaining a structured query statement, it is necessary to construct a conversion relationship between a semantic query statement and a semantic data model, and then construct a semantic translator that can convert a semantic query statement into a structured query statement based on this conversion relationship, and then input the target semantic query statement into the semantic translator for processing, and further parse the target semantic query statement into a semantic representation containing key entities, attributes, and relationships, and finally, the semantic translator gradually constructs a corresponding structured query statement based on these semantic representations and in combination with a preset conversion relationship. In practical applications, this process accurately converts a semantic query statement into a structured query statement, greatly improving the accuracy of subsequent queries.

[0228] S113: Input the structured query statement into the database for query to obtain query results.

[0229] In this embodiment, the obtained structured query statement is input into the database, and then the database executes the structured query statement to obtain the query data, and then different data charts are selected according to the data type of the query data, and finally the query data and data charts are visualized to obtain the query results.

[0230] In actual applications, entering structured query statements into the database for query can not only quickly obtain accurate data results, but also intelligently match data charts according to data types to achieve visual display of data. This process makes the data results more intuitive and easy to understand, which helps users make decisions.

[0231] In this embodiment, the application performs word segmentation processing on natural language through a pre-configured industry knowledge base and a WordPiece word segmenter to construct a prompt, and uses a pre-trained model to encode the prompt, which can accurately capture the semantic information in natural language and improve the natural language analysis and processing capabilities. Then, the first classification tag token obtained after encoding is used as the entire sentence representation, and all encoding vectors are processed according to the sentence representation to obtain a local semantic representation. The local semantic representation can reflect the complex semantic structure in natural language, which helps to better understand the user's input intention and provides a basis for subsequent semantic conversion. Furthermore, by mapping entities and attributes in natural language to database Schema elements to obtain a mapping relationship, this step establishes a bridge between natural language and structured data, so that unstructured natural language can be effectively organized and utilized. Finally, a semantic data model is constructed through mapping relationships and local semantic representations, and then a semantic parser is constructed by combining the semantic data model with custom rules. The constructed semantic parser improves the accuracy and efficiency of converting natural language into semantic query statements, provides a basis for the subsequent conversion of semantic query statements into structured query statements, and enhances the system's semantic understanding and analysis capabilities; by inputting the semantic query statements into the semantic corrector for correction, the semantic query statements are further optimized, ambiguities and errors in the semantic query statements are eliminated, the accuracy of subsequent query results is improved, and the natural language analysis capabilities are further improved; the target semantic query statements are translated through the semantic translator. This step converts the semantic query statements into structured query statements that can be recognized by the machine, which is convenient for database understanding and execution, improves query efficiency and accuracy, and provides users with a better query experience.

[0232] See also Figure 2 , Figure 2 Another embodiment of a data analysis method provided by the present application includes:

[0233] S201, converting the prompt into the input format of the pre-training model to obtain input_ids and attention_mask;

[0234] In this embodiment, the given prompt is preprocessed, including steps such as removing irrelevant characters and word segmentation, to obtain input requirements that meet the pre-training model. Then, the processed prompt is converted into an input format that the pre-training model can recognize, namely input_ids and attention_mask, where input_ids represents the index of each token, and attention_mask is used to indicate which positions are valid. In practical applications, the processed prompt is converted into the input_ids and attention_mask required by the pre-training model, which provides a basis for the pre-training model to encode the prompt.

[0235] S202. Input the input_ids and the attention_mask into the pre-trained model to obtain an encoding vector for each token.

[0236] In this embodiment, the input_ids and attention_mask obtained by preprocessing are used as input and input into the pre-trained model. Then, the pre-trained model searches for the corresponding vector in the vocabulary according to the token index in input_ids, and performs weighted processing on the input sequence in combination with attention_mask to focus on which positions are valid. Further, the model encodes input_ids and attention_mask, and finally outputs the encoding vector of each token.

[0237] In this embodiment, the prompt is converted into the input format of the pre-trained model to ensure that the pre-trained model can accurately understand the input data, thereby improving the efficiency and accuracy of natural language processing. After inputting the obtained input format input_ids and attention_mask into the pre-trained model, we can obtain the encoding vector of each token. These vectors capture the contextual information and semantic features, enhance the model's ability to understand the text, and also provide help for subsequent natural language analysis to obtain the intent in the natural language.

[0238] See also Figure 3 , Figure 3 Another embodiment of a data analysis method provided by the present application includes:

[0239] S301, extracting key entities in the natural language;

[0240] In this embodiment, before obtaining the key entities in the natural language, it is necessary to obtain the natural language input by the user, then identify the entities and attributes in the natural language, and then extract the key entities in the natural language based on the identified entities and attributes.

[0241] In actual applications, when a user enters a natural language to ask for detailed information about a product, the system will first obtain this input, then analyze, process, and identify the data and identify key entities, such as product name, model, color, etc. These key entities will be used to match product information in the database. In actual applications, extracting key entities in natural language can improve the accuracy of data processing, quickly locate and extract the core information in the language, and provide strong support for the subsequent acquisition of mapping relationships.

[0242] S302, querying database schema information to obtain Schema elements of the database;

[0243] Database schema information refers to metadata that describes the structure and organization of a database. In a database, database schema information defines the structure and relationships of elements such as tables, fields, relational models, views, indexes, and data integrity constraints. This information is usually stored in a data dictionary to ensure the consistency, integrity, and reliability of the database.

[0244] In this embodiment, before parsing the database schema information, it is necessary to connect to the target database, and then use the metadata query function provided by the database management system to retrieve the database schema information, and then extract the retrieved schema elements for subsequent matching. In this process, by connecting to the target database and retrieving the schema elements, the system can understand the structure and characteristics of the database, providing reliable data support for subsequent matching operations.

[0245] S303: Match the key entity with the Schema element to obtain a mapping relationship.

[0246] In this embodiment, before obtaining the mapping relationship, it is necessary to search in the extracted Schema elements according to the key entities in the natural language, and then match the key entities with the found Schema elements according to the search results to obtain the mapping relationship between the two. This mapping relationship enables the system to understand the user's query intention more accurately and provides support for the subsequent construction of the semantic data model.

[0247] In this embodiment, firstly, by extracting key entities in natural language, the core content of the natural language input by the user can be identified, which provides a basis for the subsequent matching of information in the database. Then, by obtaining the Schema elements in the database, the structure of the database can be accurately understood, which provides assistance for the subsequent matching of natural language to database fields. The key entities are further matched with the Schema elements to obtain the mapping relationship, which not only enhances the accuracy of subsequent queries, but also enhances the connection between natural language and database data.

[0248] See also Figure 4 , Figure 4 Another embodiment of a data analysis method provided by the present application includes:

[0249] S401, constructing an initialization semantic modifier;

[0250] In this embodiment, when constructing the initialization semantic corrector, it is necessary to define pattern matching rules, logical consistency rules and semantic consistency rules, among which the pattern matching rules are used to check whether the fields in the query statement are consistent with the database schema, the logical consistency rules are used to ensure whether the logical structure of the query statement conforms to the rules of the structured query language, and the semantic consistency rules are used to ensure whether the query statement matches the input natural language query.

[0251] S402, determining whether the semantic query statement complies with a pre-configured correction rule;

[0252] In this embodiment, the acquired semantic query statements are matched one by one with the pattern matching rules, logical consistency rules and semantic consistency rules. If the semantic query semantics conform to the pattern matching rules, logical consistency rules and semantic consistency rules, step S403 is executed; if the semantic query semantics do not conform to one or more of the rules, step S404 is executed.

[0253] S403, outputting a target semantic query statement;

[0254] In this embodiment, if the semantic query statement conforms to the pre-configured correction rule, no semantic correction is required, and the semantic corrector can directly output the semantic query statement.

[0255] S404: input the semantic query statement into a preconfigured machine learning model to obtain correction suggestions;

[0256] A machine learning model is an algorithm or set of algorithms that learns from training data, identifies patterns, relationships, or regularities in the data, and makes predictions or classifies future data based on this learned knowledge.

[0257] In this embodiment, before obtaining correction suggestions, it is necessary to build an initial machine learning model and a training data set, wherein the training data set is constructed through a large number of query statements and corresponding correction results, and then use the supervised learning method combined with the training data set to train the machine learning model to identify errors in the query statements and output corresponding correction suggestions. Further, the output correction suggestions are compared with the preset correction suggestions to obtain comparison results, and then the machine learning model is optimized according to the comparison results to obtain the target machine learning model. Finally, the semantic query statement is input into the target machine learning model to obtain the correction suggestions.

[0258] S405: Determine whether there are more than three query conditions in the semantic query statement;

[0259] In this embodiment, to determine whether there are more than three query conditions in the semantic query statement, it is necessary to identify all query conditions in the semantic query statement, then calculate the total number of query conditions in the semantic query statement, and further compare the total number of query conditions obtained with the set judgment condition. If there are no more than three query conditions, step S406 is executed, and if there are more than three query conditions, step S407 is executed. In addition to the above judgment conditions, you can also set them according to the actual situation, and no specific limitation is made here.

[0260] S406, semantically correcting the semantic query statement according to the correction suggestion to obtain a corrected query statement;

[0261] In this embodiment, if the query conditions in the semantic query statement do not exceed three, the semantic query statement is directly semantically modified in combination with the modification suggestions given by the machine learning model to obtain a modified semantic query statement.

[0262] S407, decomposing the semantic query statement to obtain a single query condition statement;

[0263] In this embodiment, a semantic query statement with more than three query conditions is parsed to obtain the keywords and conditions therein, and then the keywords and conditions are parsed and grouped according to the structure of the semantic query statement and the relationship between the semantics to form a single independent query condition. For example, the original query statement is "find mobile phones with a price lower than 100 yuan and a rating higher than 4 stars". Through the above steps, it is decomposed into two query condition statements: "mobile phones with a price lower than 100 yuan" and "mobile phones with a rating higher than 4 stars".

[0264] S408, performing semantic correction on the single query condition statement according to the correction suggestion to obtain a corrected query statement;

[0265] In this embodiment, before obtaining the corrected query statement, it is necessary to correct the obtained individual query condition statements one by one according to the correction suggestions output by the preconfigured machine learning model, and then summarize the corrected individual query condition statements to obtain the corrected query statement. This process can automatically discover and correct potential errors or unreasonableness in individual query condition statements, thereby avoiding the tediousness and error-proneness of manual correction, speeding up the correction speed of query statements, and improving the reliability of query results.

[0266] S409, evaluating the query statement to obtain an evaluation result;

[0267] In this embodiment, before obtaining the evaluation effect, it is necessary to calculate the accuracy, recall and other indicators of the query results, measure the performance improvement of the query statement based on these indicators, and then use qualitative methods, such as manually reviewing the query results and logs, to evaluate the semantic accuracy and logical rationality of the query statement, and finally combine the indicators and qualitative evaluation results to obtain the overall evaluation effect of the modified query statement. This comprehensive evaluation method helps to ensure the optimization effect of the query statement and provides some help for the subsequent acquisition of accurate semantic query statements.

[0268] S410, improving the preconfigured machine learning model and the correction rule according to the evaluation result to obtain a target semantic corrector;

[0269] In this embodiment, in order to obtain the target semantic corrector, it is necessary to further optimize the performance and correction rules of the pre-configured machine learning model according to the evaluation results. This process needs to be iterated until the preset correction effect is achieved, and then the target semantic corrector can be obtained.

[0270] In this embodiment, the target semantic corrector is obtained by improving the machine learning model and correction rules according to the evaluation results. This process can obtain objective feedback on the correction effect, optimize the machine learning model and correction rules, and improve the accuracy and efficiency of semantic correction, thereby ensuring that the corrected query statement can accurately reflect the user's true intention, improve the intelligence level of semantic processing, and provide users with a more accurate and efficient query experience.

[0271] S411: Input the semantic query statement into the target semantic corrector for correction to obtain a target semantic query statement.

[0272] In this embodiment, when a semantic query statement is input into the target semantic corrector, the semantic corrector will determine whether it complies with the internally preconfigured correction rules. If not, it will be input into the preconfigured machine learning model to obtain correction suggestions for the semantic query statement. Then it will be determined whether there are more than three query conditions in the semantic query statement. If there are more than three query conditions, the semantic query statement will be directly decomposed into statements of single query conditions, and the single query condition statements will be corrected and integrated according to the correction suggestions to obtain the target semantic query statement. If there are no more than three query conditions, the semantic query statement will be directly corrected according to the correction suggestions to obtain the target semantic query statement.

[0273] In this embodiment, by determining whether the query statement conforms to the preconfigured correction rules, the statements that need to be optimized can be quickly screened out. For semantic query statements that do not conform to the correction rules, a preconfigured machine learning model is used to analyze and process them to obtain correction suggestions, thereby enhancing the intelligent correction capability of the semantic corrector. Next, by decomposing the semantic query statements that do not conform to the correction rules and have more than three internal query conditions, a single query condition statement is obtained, which reduces the complexity of the semantic query statement and improves the correction processing efficiency. Furthermore, the single query condition statement is corrected according to the correction suggestion, which improves the correction efficiency and the accuracy of the corrected query statement. The corrected query statement is evaluated and the evaluation effect is obtained to ensure the quality of the modified target semantic query statement, providing a feedback mechanism for improving the semantic corrector.

[0274] See also Figure 5 , Figure 5 Another embodiment of a data analysis method provided by the present application includes:

[0275] S501, judging whether there is a statement error in the structured query statement according to a rule engine;

[0276] A rule engine is a software system that uses predefined rules to evaluate data and perform actions. These rules are usually written in a special language called a rule language.

[0277] In this embodiment, before determining whether there is a statement error in the structured query statement, it is necessary to construct a rule engine, which includes SQL grammar rules, database mode rules, and logical consistency rules, to ensure the accuracy of the grammar, structure, and logic of the structured query statement. Then, the structured query statement is input into the constructed rule engine for judgment. If it is determined that the structured query statement satisfies one or more rules in the rule engine, it means that there is no statement error in the structured query statement, and then step S502 is executed. If it is determined that the structured query statement does not satisfy one or more rules in the rule engine, it means that there is a statement error in the structured query statement, and then step S503 is executed.

[0278] S502, outputting a target structured query statement;

[0279] In this embodiment, if the structured query statement complies with the SQL syntax rules, database model rules, and logical consistency rules set by the rule engine, and no errors or exceptions are identified, the rule engine directly uses the structured query statement as the target structured query statement for subsequent data query.

[0280] S503: input the structured query statement into a preconfigured deep learning model to obtain a statement error type;

[0281] The deep learning model is a machine learning model based on artificial neural networks. Its core idea is to learn the feature representation and pattern of data through a multi-level neural network structure, so as to achieve understanding and prediction of data.

[0282] In this embodiment, before inputting the structured query statement into the preconfigured deep learning model, it is necessary to use a deep learning algorithm to build a model, and collect a large number of features and annotation data sets in the structured query statement, and then use the features and annotation data sets in the structured query statement to train the model, wherein the annotation data set records the error conditions and error types of each structured query statement, and further, the model can be trained to obtain a deep learning model that can identify and classify statement errors. After the deep learning model is built, the structured query statement can be input into the deep learning model, and the deep learning model will output the statement error type corresponding to the structured query statement, such as syntax error, logic error, missing field, etc.

[0283] S504, determining whether the statement error type is a field name error;

[0284] In this embodiment, the system will judge based on the acquired statement error type to check whether the statement error type is a field name error. If the statement error type is a field name error, step S505 is executed, and if the statement error type is not a field name error, step S506 is executed.

[0285] S505, adjusting the statement error to obtain a target structured query statement;

[0286] In this embodiment, if the statement error type is a field name error, the system will automatically provide similar field names based on the semantics and content in the structured query statement, and then perform statement correction based on the similar field names to obtain the target structured query statement.

[0287] S506, sending a prompt selection to the user to obtain a selection result;

[0288] In this embodiment, if the statement error type is not a field name error, the system will ask the user through the interface and provide the user with choices, for example, "Do you want to use field A or field B to query?", and then obtain the user's selection result through the user's selection and confirmation.

[0289] S507: Adjust the structured query statement according to the selection result to obtain a target structured query statement.

[0290] In this embodiment, the structured query statement is analyzed according to the user's selection results, and the parts that need to be modified are identified to obtain modification suggestions. Then, the structured query statement is modified according to the modification suggestions, for example, the internal query conditions or sorting are modified. After the modification is completed, the target structured query statement can be obtained.

[0291] In this embodiment, by using a rule engine and a machine learning model to identify and classify structured query statements, the system can automatically identify potential errors and error types in structured query statements, and improve the accuracy and efficiency of detection. By determining the error type of the statement, the error of the structured query statement can be quickly located, unnecessary detection steps can be reduced, and the speed and accuracy of problem solving can be improved. If the statement error type is not a field error, a prompt request is issued to the user. This step is used to issue a prompt selection to the user when the best adjustment direction cannot be determined. Adjusting the query statement according to the user's selection result semantically conforms to the user's query intent, realizes personalized query, and ensures the accuracy of the query statement.

[0292] See also Figure 6 , Figure 6 Another embodiment of a data analysis method provided by the present application includes:

[0293] S601, inputting the target structured query statement into the database for query to obtain query data;

[0294] In this embodiment, when performing a database query, it is necessary to establish a connection between the system and the database, and then input the adjusted target structured query statement into the target database. Then, the database retrieves all the data in the database according to the query conditions in the target structured query statement. Further, the database sorts all the data that meets the query conditions to obtain the query data. For example, the obtained query statement "select * from customers where age>30" is input into the database, and then the database retrieves all customer data older than 30 years old according to this query statement instruction. Further, the database sorts and adjusts these customer data and returns a query data containing these customer data.

[0295] S602, extracting statistical features of the query data;

[0296] In this embodiment, before extracting the statistical characteristics of the query data, the mean in the query data is first calculated. Then the query data is sorted from small to large, and the position of the median is determined according to the parity of the number of query data. Then the degree of dispersion of the query data, i.e., the variance, is calculated. Further, the asymmetry of the query data distribution, i.e., the skewness, is calculated. Further, the sharpness of the query data release, i.e., the kurtosis, is evaluated. Finally, the obtained mean, median, variance, skewness, and kurtosis are collected and sorted to obtain the statistical characteristics of the query data.

[0297] S603, inputting the statistical features into a preconfigured recognition model to obtain a distribution type;

[0298] In statistics, distribution type refers to the form or distribution law that data or random variables may present. Different distribution types have different characteristics and application scenarios, and can describe and characterize the probability distribution characteristics of data under different circumstances. Common distribution types include normal distribution, uniform distribution and skewed distribution.

[0299] In this embodiment, before inputting the statistical features into the preconfigured recognition model, a recognition model needs to be constructed. The recognition model can be constructed by using a decision tree, a random forest or a deep learning algorithm. Then, a data set is constructed by collecting multiple data distribution types and corresponding visualization methods. Further, the data set and the statistical features are input into the recognition model for training to obtain a target recognition model that can identify the data distribution type. Among them, the target recognition model is mainly used to analyze the statistical features, and automatically identify and output the corresponding distribution type according to the analysis results.

[0300] S604, generating a data chart according to the distribution type;

[0301] Data charts are a way of organizing and presenting data using visual elements such as graphics, tables, and symbols. They encode and transform data so that it can be presented to the audience in a more intuitive and easy-to-understand form, thereby helping people better analyze, understand, and communicate data.

[0302] In this embodiment, it is necessary to identify whether the obtained distribution type is normal distribution, uniform distribution or skewed distribution, and then map the identified distribution type to a visualization chart to finally generate a data chart. Taking normal distribution as an example, if the distribution type is normal distribution, the system recommends selecting a histogram or a density map.

[0303] S605: Visually display the query data and the data chart to obtain query results.

[0304] Visual presentation uses techniques and methods such as graphic design, data visualization, information graphics and interactive design to transform complex data, information or knowledge into an easy-to-understand and intuitive visual form.

[0305] In this embodiment, the query data obtained from the database and the generated data chart need to be integrated, and then the integrated result is displayed on the user interface in a visual display manner, and the displayed result is the query result.

[0306] In this embodiment, by executing the target structured query statement in the database, query data that meets the query conditions can be obtained. Then, the statistical features of these query data are extracted, and the statistical features are input into the pre-configured recognition model. The recognition model can automatically determine the distribution type of the data, reducing the errors of manual operation. Further, according to the identified distribution type, the corresponding data chart is automatically generated to enhance data visualization. Finally, the query data and the data chart are combined and visualized, providing users with an intuitive data analysis result, enabling users to make more informed decisions based on these intuitive data, further improving the ability of data processing.

[0307] See also Figure 7 , Figure 7 A data analysis system provided for this application comprises:

[0308] An acquisition unit 701 is used to acquire a natural language input by a user;

[0309] A first construction unit 702 is used to construct an initial data model;

[0310] The second construction unit 703 is used to perform word segmentation processing on the natural language according to the preconfigured industry knowledge base and the WordPiece word segmenter to construct a prompt;

[0311] An encoding unit 704, configured to encode the prompt using a pre-trained model to obtain an encoding vector for each token;

[0312] A first processing unit 705 is used to represent the entire sentence using the encoding vector of the first classification tag token output by the pre-training model;

[0313] A second processing unit 706 is used to process all the encoding vectors according to the sentence representation to obtain a local semantic representation;

[0314] The third processing unit 707 is used to identify entities and attributes in the natural language and map the entities and attributes to Schema elements of the database to obtain a mapping relationship;

[0315] A third construction unit 708 is used to construct a semantic data model according to the mapping relationship and the local semantic representation;

[0316] A fourth construction unit 709 is used to construct a semantic parser by combining the semantic data model with a custom rule;

[0317] The fourth processing unit 710 is used to input the natural language into the semantic parser for parsing to obtain a semantic query statement;

[0318] A fifth processing unit 711 is used for inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement;

[0319] A sixth processing unit 712, configured to convert the target semantic query statement into a structured query statement through a semantic translator;

[0320] The query unit 713 is used to input the structured query statement into the database for query to obtain a query result.

[0321] Optionally, the encoding unit 704 is further configured to:

[0322] Convert the prompt into the input format of the pre-trained model to obtain input_ids and attention_mask;

[0323] The input_ids and the attention_mask are input into the pre-trained model to obtain the encoding vector of each token.

[0324] Optionally, the third processing unit 707 is further configured to:

[0325] Extracting key entities in the natural language;

[0326] Query database schema information to obtain the Schema element of the database;

[0327] The key entity is matched with the Schema element to obtain a mapping relationship.

[0328] Optionally, the fifth processing unit 711 is further configured to:

[0329] Construct initialization semantic modifier;

[0330] Determining whether the semantic query statement complies with a preconfigured correction rule;

[0331] If not, input the semantic query statement into a preconfigured machine learning model to obtain correction suggestions;

[0332] Determining whether there are more than three query conditions in the semantic query statement;

[0333] If yes, decomposing the semantic query statement to obtain a single query condition statement;

[0334] Performing semantic correction on the single query condition statement according to the correction suggestion to obtain a corrected query statement;

[0335] Evaluate the query statement and obtain evaluation results;

[0336] Improving the preconfigured machine learning model and the correction rule according to the evaluation effect to obtain a target semantic corrector;

[0337] The semantic query statement is input into the target semantic corrector for correction to obtain a target semantic query statement.

[0338] Optionally, a seventh processing unit 714 is further included, configured to:

[0339] Determine whether there is a statement error in the structured query statement according to the rule engine;

[0340] If yes, input the structured query statement into a preconfigured deep learning model to obtain the statement error type;

[0341] Determine whether the statement error type is a field name error;

[0342] If not, a prompt selection is issued to the user to obtain a selection result;

[0343] The structured query statement is adjusted according to the selection result to obtain a target structured query statement.

[0344] Optionally, the query unit 713 is further configured to:

[0345] Inputting the target structured query statement into the database for query to obtain query data;

[0346] Selecting a data chart according to the data type of the query data;

[0347] The query data and the data chart are visually displayed to obtain query results.

[0348] Optionally, the query unit 713 is further configured to:

[0349] Extracting statistical features of the query data;

[0350] Inputting the statistical features into a preconfigured recognition model to obtain a distribution type;

[0351] A data chart is generated according to the distribution type.

[0352] Optionally, the third building block is specifically used for:

[0353] Initialize the mapping relationship based on the context information of the currently input natural language query

[0354] Adjust the weight of the Schema element of the database according to the context information

[0355] Constructing an initial semantic graph, wherein the initial semantic graph includes entity nodes, relationship nodes and operation nodes, entity relationship edges, conditional relationship edges and operation relationship edges;

[0356] For each node, a comprehensive node feature representation is generated by combining the local semantic representation, the metadata information of the database, and the user's historical query behavior.

[0357] Initialize the features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation

[0358] Embed the comprehensive node feature representation and the comprehensive edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on a sink algorithm that implements a connector to obtain an input semantic graph;

[0359] The input semantic graph is input into a pre-built initial graph neural network for training, the information of neighbor nodes is aggregated through the graph convolution operation of each layer in the initial graph neural network, and the status of each node is updated in combination with the characteristics of each node;

[0360] Using the Adam optimizer to optimize the initial graph neural network to obtain a converged semantic data model;

[0361] Optionally, the fourth building block is specifically used for:

[0362] Constructing a standardized query template framework, wherein the standardized query template framework includes selection operations, filtering conditions, sorting and grouping operations;

[0363] Build entity mapping rules to map entities in natural language queries to tables and columns in the semantic data model;

[0364] Build condition mapping rules to map conditions in natural language queries to standard operators in the database;

[0365] Build sorting and grouping rules to process sorting and grouping operations in natural language queries, mapped to ORDER BY and GROUP BY in SQL statements;

[0366] Build subquery rules to process complex queries containing subqueries and generate corresponding query commands;

[0367] Build join rules to process join operations between multiple tables and generate corresponding SQL join conditions;

[0368] A filling relationship is constructed to embed the output of the custom rule into the standardized query template framework to obtain a semantic parser.

[0369] See also Figure 8 , the present application also provides another data analysis system, the system comprising:

[0370] Processor 801, memory 802, input and output unit 803, bus 804;

[0371] The processor 801 is connected to the memory 802, the input and output unit 803 and the bus 804;

[0372] The memory 802 stores a program, and the processor 801 calls the program to execute any of the above methods.

[0373] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0374] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0375] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0376] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0377] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0378] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A data analysis method, characterized in that: The method comprises: Get the natural language input by the user; Build the initial data model; Perform word segmentation on the natural language according to a preconfigured industry knowledge base and a WordPiece word segmenter to construct a prompt; Encode the prompt through the pre-trained model to obtain the encoding vector of each token; The encoding vector of the first classification tag token output by the pre-trained model is used as the entire sentence representation; Processing all the encoding vectors according to the sentence representation to obtain a local semantic representation; By identifying entities and attributes in the natural language, and mapping the entities and attributes to Schema elements of a database, a mapping relationship is obtained; Constructing a semantic data model according to the mapping relationship and the local semantic representation; Combining the semantic data model with custom rules to build a semantic parser; Inputting the natural language into the semantic parser for parsing to obtain a semantic query statement; Inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement; Converting the target semantic query statement into a structured query statement by a semantic translator; Inputting the structured query statement into the database for query to obtain query results; The constructing of the semantic data model according to the mapping relationship and the local semantic representation includes: Initializing the mapping relationship according to context information of the currently input natural language query; Adjusting the weight of the Schema element of the database according to the context information; Constructing an initial semantic graph, wherein the initial semantic graph includes entity nodes, relationship nodes and operation nodes, entity relationship edges, conditional relationship edges and operation relationship edges; For each node, combining the local semantic representation, metadata information of the database, and historical query behavior of the user to generate a comprehensive node feature representation of each node; Initializing features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation; Embed the comprehensive node feature representation and the comprehensive edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on a sink algorithm that implements a connector to obtain an input semantic graph; The input semantic graph is input into a pre-built initial graph neural network for training, the information of neighbor nodes is aggregated through the graph convolution operation of each layer in the initial graph neural network, and the status of each node is updated in combination with the characteristics of each node; The initial graph neural network is optimized using the Adam optimizer to obtain a converged semantic data model.

2. The data analysis method according to claim 1, characterized in that: The prompt is encoded through the pre-trained model to obtain the encoding vector of each token, including: Convert the prompt into the input format of the pre-trained model to obtain input_ids and attention_mask; The input_ids and the attention_mask are input into the pre-trained model to obtain the encoding vector of each token.

3. The data analysis method according to claim 1, characterized in that: By identifying entities and attributes in the natural language, and mapping the entities and attributes to Schema elements of the database, a mapping relationship is obtained, including: Extracting key entities in the natural language; Query database schema information to obtain the Schema element of the database; The key entity is matched with the Schema element to obtain a mapping relationship.

4. The data analysis method according to claim 1, characterized in that: Inputting the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement, including: Construct initialization semantic modifier; Determining whether the semantic query statement complies with a preconfigured correction rule; If not, input the semantic query statement into a preconfigured machine learning model to obtain correction suggestions; Determining whether there are more than three query conditions in the semantic query statement; If yes, decomposing the semantic query statement to obtain a single query condition statement; Performing semantic correction on the single query condition statement according to the correction suggestion to obtain a corrected query statement; Evaluate the query statement and obtain evaluation results; Improving the preconfigured machine learning model and the correction rule according to the evaluation effect to obtain a target semantic corrector; The semantic query statement is input into the target semantic corrector for correction to obtain a target semantic query statement.

5. The data analysis method according to claim 1, characterized in that: After converting the target semantic query statement into a structured query statement by a semantic translator, the method further includes: Determine whether there is a statement error in the structured query statement according to the rule engine; If yes, input the structured query statement into a preconfigured deep learning model to obtain the statement error type; Determine whether the statement error type is a field name error; If not, a prompt selection is issued to the user to obtain a selection result; The structured query statement is adjusted according to the selection result to obtain a target structured query statement.

6. The data analysis method according to claim 5, characterized in that: Inputting the structured query statement into the database to perform a query to obtain a query result includes: Inputting the target structured query statement into the database for query to obtain query data; Selecting a data chart according to the data type of the query data; The query data and the data chart are visually displayed to obtain query results.

7. The data analysis method according to claim 6, characterized in that: Selecting a data chart according to the data type of the query data includes: Extracting statistical features of the query data; Inputting the statistical features into a preconfigured recognition model to obtain a distribution type; A data chart is generated according to the distribution type.

8. A data analysis system, characterized in that: The system comprises: An acquisition unit, used for acquiring natural language input by a user; A first construction unit is used to construct an initial data model; The second construction unit is used to perform word segmentation processing on the natural language according to a preconfigured industry knowledge base and a WordPiece word segmenter to construct a prompt; An encoding unit, used to encode the prompt through a pre-trained model to obtain an encoding vector for each token; A first processing unit, configured to use the encoding vector of the first classification tag token output by the pre-trained model as the entire sentence representation; A second processing unit, configured to process all the encoding vectors according to the sentence representation to obtain a local semantic representation; A third processing unit is used to identify entities and attributes in the natural language and map the entities and attributes to Schema elements of a database to obtain a mapping relationship; A third construction unit, configured to construct a semantic data model according to the mapping relationship and the local semantic representation; A fourth construction unit, configured to construct a semantic parser by combining the semantic data model with a custom rule; A fourth processing unit, configured to input the natural language into the semantic parser for parsing to obtain a semantic query statement; A fifth processing unit, configured to input the semantic query statement into a semantic corrector for correction to obtain a target semantic query statement; A sixth processing unit, configured to convert the target semantic query statement into a structured query statement through a semantic translator; A query unit, used for inputting the structured query statement into the database for querying to obtain a query result; The third building block is specifically used for: Initialize the mapping relationship based on the context information of the currently input natural language query Adjusting the weight of the Schema element of the database according to the context information; Constructing an initial semantic graph, wherein the initial semantic graph includes entity nodes, relationship nodes and operation nodes, entity relationship edges, conditional relationship edges and operation relationship edges; For each node, combining the local semantic representation, metadata information of the database, and historical query behavior of the user to generate a comprehensive node feature representation of each node; Initializing features of each edge in the initial semantic graph to obtain a comprehensive edge feature representation; Embed the comprehensive node feature representation and the comprehensive edge feature representation into the initial semantic graph, and dynamically update the initial semantic graph based on a sink algorithm that implements a connector to obtain an input semantic graph; The input semantic graph is input into a pre-built initial graph neural network for training, the information of neighbor nodes is aggregated through the graph convolution operation of each layer in the initial graph neural network, and the status of each node is updated in combination with the characteristics of each node; The initial graph neural network is optimized using the Adam optimizer to obtain a converged semantic data model.

9. A data analysis system, characterized in that: The system comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images