Data analysis method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a data analysis method and device, equipment and a storage medium, and the method comprises the steps: obtaining an original query problem, classifying the original query problem, and determining a problem type to which the original query problem belongs; identifying a data entity in the original query problem based on the problem type so as to determine a database table associated with the original query problem according to the data entity; perfecting the data entity according to the database table to determine a target query problem; generating a target query statement based on the target query problem, and determining a data query result according to the target query statement. According to the method, efficient completion of data query and visual analysis tasks can be realized through an automatic process of the intelligent agent, the efficiency of data analysis is improved, and the accuracy and reliability of a data analysis result are ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data analysis method, device, equipment and storage medium. Background Art

[0002] A data lake is a large-scale data storage and processing system that can store data of various types and formats. The purpose of a data lake is to enable enterprises to better manage and utilize large amounts of data for data analysis, machine learning, and other tasks. The traditional method of data lake data search and visualization analysis is to have different people perform demand analysis, data search, data extraction, data processing, and other steps according to user needs, resulting in a long data query and analysis cycle and low overall efficiency, and the final results are difficult to satisfy users. Summary of the invention

[0003] In view of this, the present invention provides a data analysis method, device, equipment and storage medium to solve the problem of low efficiency in data processing and analysis.

[0004] In a first aspect, the present invention provides a data analysis method, the method comprising:

[0005] Obtaining an original query question, and classifying the original query question to determine the question type to which the original query question belongs;

[0006] Identify a data entity in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity;

[0007] Completing the data entity according to the metadata of the database table to determine the target query question;

[0008] A target query statement is generated based on the target query question to determine a data query result according to the target query statement.

[0009] In some optional implementations, classifying the original query question to determine the question type to which the original query question belongs includes:

[0010] Performing vectorization processing on the original query question;

[0011] Logistic regression classification is performed on the original query question after vectorization processing to determine the question type to which the original query question belongs.

[0012] In some optional implementations, the identifying a data entity in the original query question based on the question type to determine a database table associated with the original query question according to the data entity includes:

[0013] Determine a corresponding recognition model based on the problem type;

[0014] Processing the original query question based on the recognition model to determine the data entity;

[0015] A database table associated with the original query question is queried according to the data entity.

[0016] In some optional implementations, the step of improving the data entity according to the metadata of the database table to determine the target query question includes:

[0017] Determining whether the data entity needs to be adjusted based on a preset knowledge base and metadata of the database table;

[0018] If the data entity needs to be adjusted, the data entity is improved to obtain the target query question.

[0019] In some optional implementations, generating a target query statement based on the target query question to determine a data query result according to the target query statement includes:

[0020] Filtering the metadata of the database table based on a preset knowledge base and the target query question to determine a target data field in the database table;

[0021] Generate a target query statement according to the preset knowledge base and the target query question;

[0022] A data query result is generated based on the target query statement and the target data field.

[0023] In some optional implementations, generating a target query statement according to the preset knowledge base and the target query question includes:

[0024] Transforming specific fields in the target query question based on a preset knowledge base to determine the query field;

[0025] A target query statement is generated based on the query field.

[0026] In some optional embodiments, the method further comprises:

[0027] Acquire original data resources, and preprocess the original data resources;

[0028] The preprocessed original data resources are stored in a database to obtain a preset knowledge base, which at least includes a vector database and a graph database.

[0029] The data analysis method provided in this embodiment obtains the original query question, classifies the original query question, and determines the question type to which the original query question belongs; identifies the data entity in the original query question based on the question type, so as to determine the database table associated with the original query question according to the data entity; improves the data entity according to the metadata of the database table to determine the target query question; generates a target query statement based on the target query question, so as to determine the data query result according to the target query statement. This method can realize the efficient completion of data query and visual analysis tasks through the automated process of the intelligent body, improve the efficiency of data analysis, and ensure the accuracy and reliability of the data analysis results.

[0030] In a second aspect, the present invention provides a data analysis device, the device comprising:

[0031] A type determination module is used to obtain an original query question, classify the original query question, and determine the question type to which the original query question belongs;

[0032] A subject identification module, used to identify a data entity in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity;

[0033] A problem determination module, used to improve the data entity according to the metadata of the database table to determine the target query problem;

[0034] The result determination module is used to generate a target query statement based on the target query question to determine the data query result according to the target query statement.

[0035] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data analysis method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data analysis method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a flow chart of a data analysis method according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of generating a target query statement according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of another data analysis method according to an embodiment of the present invention;

[0041] Figure 4 is a structural block diagram of a data analysis device according to an embodiment of the present invention;

[0042] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0044] Currently, enterprises are undergoing digital transformation. In the process of transformation from informatization to digitization, enterprises have accumulated massive amounts of data. Through scientific data governance, they have established enterprise data lakes to achieve data aggregation and connection, giving data a clear definition and unified structure, making data easier to obtain, and breaking data silos and monopolies. More and more enterprises and institutions in the fields of finance, petrochemicals, industrial manufacturing, retail, and medical care have begun to build enterprise-level data lakes, supporting business decisions, risk management, and innovative applications through search and visualization of data in data lakes.

[0045] In the related art, the traditional methods of data lake data search and visualization analysis are completed by different personnel respectively. First, professional data developers use special tools to search for specific data in the data lake according to data requirements, generate initial data through collection, modeling, cleaning, export and other steps, and send them to data analysts. Data analysts then use data analysis tools to import data, and then perform data visualization processing and analysis to obtain analysis results, and then submit the analysis result data to decision makers for use. From obtaining analysis requirements, searching data in the data lake, extracting data, preprocessing data, to data entering the data analysis tool for analysis and processing, and finally to the decision maker, it has gone through multiple people, multiple steps, and multiple tool software, which takes a long time. It is necessary to have an in-depth understanding of the data structure and data distribution in the data lake, have professional capabilities in data extraction, cleaning and preprocessing, and be proficient in the use of professional analysis tools. In addition, it is necessary to go through multiple processes and multiple people to complete data query analysis, and the results obtained may not be satisfactory to the end user. It may take several rounds to get satisfactory results. Based on this, the present invention provides a data analysis method, which can be applied to a data analysis system, specifically a data lake data search and visualization analysis integrated system based on a large model.

[0046] According to an embodiment of the present invention, a data analysis method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] In this embodiment, a data analysis method is provided, which can be used in terminals such as mobile phones, tablet computers, and computers. Figure 1 is a flow chart of a data analysis method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0048] Step S101, obtaining an original query question, and classifying the original query question to determine the question type to which the original query question belongs.

[0049] The data analysis system will provide different types of users with corresponding permissions, which are related to the accessible database. When users need to query and analyze data, they can enter original query questions in the data analysis system. Original query questions include data query, data question and answer, etc. The large model needs to identify the user's intention and then call the corresponding functional module.

[0050] When the user inputs the original query question, the original query question is obtained and classified. Specifically, the original query question can be classified and processed by a text classification algorithm based on machine learning, and the question type to which the original query question belongs is output. Question types include data query, data question and answer, etc.

[0051] Exemplarily, when a user queries data in a data lake, it is necessary to identify the intent of the original query question, call a corresponding classification function module, and classify the original query question to determine the question type to which the original query question belongs.

[0052] Step S102: identifying data entities in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity.

[0053] After determining the type of question to which the original query belongs, different processing models can be called according to the question type, and the subject in the original query question can be identified through the corresponding processing model. A data entity is an object or concept with a clear identity or attribute mentioned in the original query question, and is usually the core object that the user wants to query or obtain information.

[0054] For example, if the original query question is "How many employees does Company A have in 2020", after analysis, it is necessary to determine the query time, company and indicators, where the query time is "2020", the company is "Company A", and the indicator is "number of employees". In this embodiment, the data entities include "2020", "Company A" and "number of employees".

[0055] After the data entity is determined, the database tables related to the data entity are obtained from the data lake. These database tables can be considered as database tables associated with the original query question.

[0056] Step S103: perfecting the data entity according to the metadata of the database table to determine the target query question.

[0057] After the original query question is identified, the knowledge retrieval module can be used to obtain highly relevant, timely and relatively rich content based on the original query question. First, the original query question needs to be rewritten to a searchable query question with clearer semantics and richer meaning. Based on the query subject and parameter identification, information comparison is performed in combination with the knowledge base information to complete the data entities that need to be queried and correct information errors.

[0058] In some optional implementations, before the data entity is improved, it is necessary to determine whether the data entity needs to be improved through information comparison. During the information comparison, the data entity can be compared with the metadata in the database table respectively.

[0059] Step S104: generating a target query statement based on the target query question, so as to determine a data query result according to the target query statement.

[0060] Generate a corresponding target query statement according to the target query question, the target query statement is a database query statement, for example, it can be an SQL statement. In some optional implementations, if the data entity involved in the target query question is not a field in the database table, it is necessary to obtain data from the database table before calculating the data entity. In this case, it is necessary to rewrite it in a targeted manner, and output the target query statement by calling the large language model.

[0061] In some optional implementations, in order to improve the efficiency of knowledge base data retrieval, before generating the target query statement, the database metadata information can be filtered according to the target query question, specifically including filtering metadata attributes such as databases, tables, fields, etc., filtering out irrelevant metadata information, thereby improving efficiency and relevance.

[0062] After generating the target query statement, data query is performed according to the target query statement, and data plotting tasks are executed according to the data type. According to the planning results of the task by the large model, the data plotting module is called to execute the data plotting task. The plotting type specified by the user (line chart, bar chart, pie chart, scatter plot, perspective chart, table, etc.) is automatically selected, and the analysis result data is mapped to the attributes of the graphic to generate the corresponding image code and display it, which is the data query result.

[0063] The data analysis method provided in this embodiment obtains the original query question, classifies the original query question, and determines the question type to which the original query question belongs; identifies the data entity in the original query question based on the question type, so as to determine the database table associated with the original query question according to the data entity; improves the data entity according to the metadata of the database table to determine the target query question; generates a target query statement based on the target query question, so as to determine the data query result according to the target query statement. This method can realize the efficient completion of data query and visual analysis tasks through the automatic calling of the large language model process by the intelligent body, improve the efficiency of data analysis, and ensure the accuracy and reliability of the data analysis results.

[0064] In this embodiment, a data analysis method is provided, which can be used in terminals such as mobile phones, tablet computers, and computers. The method includes the following steps:

[0065] Step S201: obtain an original query question, classify the original query question, and determine the question type to which the original query question belongs.

[0066] Specifically, the above step S201 includes:

[0067] Step S2011, vectorizing the original query question;

[0068] Step S2012, performing logistic regression classification processing on the original query question after vectorization processing to determine the question type to which the original query question belongs.

[0069] In order to understand and process the original query questions entered by the user, it is first necessary to convert the query text into a vector representation. A pre-trained Chinese text vector representation model can be used to convert the text into a high-dimensional, standardized vector.

[0070] After obtaining the vector representation of the original query question, the vector is used as the input of the logistic regression model. The logistic regression model is a linear classifier that can predict the category to which the new input vector belongs by learning the relationship between the input vector and the classification label. In this embodiment, the classification label may include data query, metadata query, data question and answer, etc., and corresponding classification labels will be prepared in advance for each possible user intention. After the logistic regression classification processing of the model, the probability distribution of the original query question is output, indicating the probability that the original query question belongs to each category. Based on the output results, the system can select the category with the highest probability as the question type to which the original query question belongs.

[0071] In an optional implementation, the text2vec-large-chinese model can be used as an embedding model, wherein text2vec-large-chinese is a pre-trained model for vector representation of Chinese text. In the field of natural language processing, the embedding model is usually used to convert text (such as words, sentences or paragraphs) into vector representations in a high-dimensional space. The embedding model is called to convert the input text into a standardized high-dimensional vector, and then the high-dimensional vector is used as input for logistic regression classification.

[0072] Step S202: identifying data entities in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity.

[0073] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0074] Step S203: perfecting the data entity according to the metadata of the database table to determine the target query question.

[0075] For details, please see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0076] Step S204: Generate a target query statement based on the target query question to determine the data query result according to the target query statement. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0077] The data analysis method provided in this embodiment, combined with a deep learning model and a traditional machine learning algorithm, can effectively handle the classification task of user queries. By converting the original query problem into a vector representation and then performing a logistic regression classification method for classification, it is possible to accurately identify the user's query intention and perform subsequent data analysis.

[0078] In this embodiment, a data analysis method is provided, which can be used in terminals such as mobile phones, tablet computers, and computers. The method includes the following steps:

[0079] Step S301: obtain an original query question, classify the original query question, and determine the question type to which the original query question belongs.

[0080] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0081] Step S302: identifying data entities in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity.

[0082] Specifically, the above step S302 includes:

[0083] Step S3021, determining a corresponding recognition model based on the question type;

[0084] Step S3022, processing the original query question based on the recognition model to determine the data entity;

[0085] Step S3023: query the database table associated with the original query question according to the data entity.

[0086] For different question types, the corresponding recognition model will be called for processing to identify the entities in the original query question (Query). Question types include data query, metadata query, data question and answer, etc. The corresponding recognition model can perform text processing on questions of this type of sentence structure in a targeted manner to extract the data entities therein. Taking data question and answer as an example, the original query question may be "How many types of xx tables are there in total?", and the extracted data entity is "xx table".

[0087] For the extracted data entity, the table and related fields related to the data entity are obtained from the data lake. All related tables queried based on the data entity are collectively referred to as database tables in this embodiment.

[0088] When extracting data entities, a keyword recall method can be used to construct a full-text index, segment the original query, and use an information retrieval algorithm (for example, the BM25 information retrieval algorithm) for retrieval.

[0089] Step S303: perfecting the data entity according to the metadata of the database table to determine the target query question.

[0090] For details, please see Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0091] Step S304: generating a target query statement based on the target query question, so as to determine a data query result according to the target query statement.

[0092] For details of step S304, see Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0093] The data analysis method provided in this embodiment calls the corresponding recognition model based on the question type, so as to accurately identify the data entities in the original query question. This targeted processing method improves the accuracy and efficiency of data entity recognition.

[0094] In this embodiment, a data analysis method is provided, which can be used in terminals such as mobile phones, tablet computers, and computers. The method includes the following steps:

[0095] Step S401: obtain an original query question, classify the original query question, and determine the question type to which the original query question belongs.

[0096] For details, please see Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0097] Step S402: identifying data entities in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity.

[0098] For details, please see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0099] Step S403: perfecting the data entity according to the metadata of the database table to determine the target query question.

[0100] Specifically, the above step S403 includes:

[0101] Step S4031, judging whether the data entity needs to be adjusted based on the metadata of the preset knowledge base and database table;

[0102] Step S4032: If the data entity needs to be adjusted, then improve the data entity to obtain the target query problem.

[0103] Before improving the data entity, it is first necessary to determine whether the data entity needs to be adjusted. Specifically, it is judged through a preset knowledge base and database tables. The preset knowledge base is a database including information such as industry knowledge, data standard information, and metadata information. The extracted data entity is compared with the fields in the database tables and the preset knowledge base, so as to determine whether the data entity needs to be adjusted.

[0104] As an example, if the extracted data entity includes "Company A", after information comparison, it is found that the full name of "Company A" is "A Limited Liability Company", indicating that the data entity needs to be adjusted and modified to "A Limited Liability Company". After comparing and adaptively improving each data entity in the original query problem one by one, the target query problem is obtained.

[0105] In some optional implementation manners, the target query problem can also be improved by the method of knowledge graph recall from the knowledge graph information (relationship information) stored in the preset knowledge base, so as to improve the accuracy of the final data analysis result. For example, if the original query problem is to query in which cities Company A has set up branch offices, the following information can be recalled through the knowledge graph:

[0106] <Company A, has set up a branch office, City B>;

[0107] <Company A, has set up a branch office, City C>;

[0108] <Company A, has set up a branch office, City D>.

[0109] These information are added to the original query problem as supplementary information to obtain the improved target query problem.

[0110] Step S404: Generate a target query statement based on the target query problem to determine the data query result according to the target query statement.

[0111] Specifically, the above step S404 includes:

[0112] Step S4041: Filter the metadata in the database table based on the preset knowledge base and the target query problem to determine the target data fields in the database table.

[0113] In order to improve the efficiency of knowledge base data retrieval, the data in the database table is filtered according to the target query question. Specifically, the metadata attributes such as database, table, and field can be filtered to improve efficiency and relevance. After filtering, the target data field related to the target query question is obtained. For example, if the original query question is "How many employees does A Co., Ltd. have in 2020", you can first identify which tables and fields the user query will involve based on the metadata information in the preset knowledge base, filter out irrelevant metadata information, and further query rewrite only for these tables and fields.

[0114] Step S4042, generating a target query statement according to a preset knowledge base and a target query question;

[0115] After determining the target query problem, the target query statement can be output in combination with the preset knowledge base. Taking the target query statement as an SQL statement as an example, the NL2SQL (Text-to-SQL) module is called for data query and calculation. Among them, the NL2SQL module can convert natural language queries into SQL queries that can be executed on a relational database.

[0116] Step S4043, generating data query results based on the target query statement and the target data field.

[0117] Based on the target query statement, the target data field is queried to obtain relevant data results. During the query process, a large amount of context can be retrieved using the multi-recall method, but not all contexts are relevant to the question. For example, a reranking model (e.g., a bge-reranker-base model) can be used to rearrange the order of the recalled context content, exclude irrelevant or unimportant content, and put relevant content at the front to improve query efficiency.

[0118] For the queried data results, the drawing module is called to perform the corresponding data drawing task. The relevant graph types are called according to the type of data results. The graph types include line graphs, bar graphs, pie charts, scatter graphs, perspective graphs, tables, etc. The data query results are displayed to the user in a visual form.

[0119] In some optional implementations, step S4042 includes: transforming a specific field in the target query question based on a preset knowledge base to determine a query field; and generating a target query statement based on the query field.

[0120] In scenarios involving data statistics or indicator calculations, it is necessary to generate the required specific fields according to the target query problem for conversion, so that the specific fields are converted into query fields. The query fields represent the existing fields in the database table, and the specific fields represent the data that can only be obtained based on the processing of the specific fields.

[0121] For example, Figure 2 As shown, when financial indicator calculation is required, the corresponding calculation formula is recalled from the pre-established financial calculation indicator formula library through SQL recall, where the specific field is asset turnover rate. By querying the preset knowledge base, the calculation formula of asset turnover rate can be obtained as sales revenue / total assets. Sales revenue and total assets are query fields. There is a database table related to sales revenue and total assets, and the target query statement is generated based on the conversion.

[0122] In some optional implementations, the data analysis method further includes: obtaining original data resources and preprocessing the original data resources; storing the preprocessed original data resources in a database to obtain a preset knowledge base. The preset knowledge base includes at least a vector database and a graph database.

[0123] The system needs to build a preset knowledge base in advance, which includes information extraction and index storage for diversified data, data preprocessing, document segmentation, document tree construction, embedding (the technology of mapping high-dimensional data to low-dimensional vector space) and data storage. The original data resources include industry knowledge, data standard information, metadata information, data resource directory related to the data lake, and the results of the data governance process (business terms, business objects, object relationships). Among them, data standard information includes Chinese translation of English fields, field explanations, and field types.

[0124] Different preprocessing can be performed for different types of data. For example, for raw data resources such as industry knowledge, data standard information (Chinese, explanation, type of fields), and metadata information, the data is converted into vector form through the embedding model after preprocessing and stored in the vector database. For data representing the relationship between data entities such as the data resource directory related to the data lake and the results of the data governance process, an industry knowledge graph is constructed through the graph tool after preprocessing and stored in the graph database.

[0125] In this embodiment, a specific data analysis method is provided, such as Figure 3 As shown in the figure, it includes steps such as knowledge base construction, intent recognition, knowledge retrieval, NL2SQL and execution analysis, and data mapping. Among them, the knowledge retrieval step includes query rewriting, metadata filtering, multi-level recall, and relevance re-ranking. In the NL2SQL and execution analysis step, the NL2SQL model is used to generate query statements.

[0126] This method uses the ReAct architecture to develop an intelligent agent for data lake query and analysis. The intelligent agent calls the large language model and uses the reasoning ability, generation ability and tool use ability of the large language model to convert the natural language requirements input by the end user into data query analysis and data visualization tasks, and automatically completes the automatic query analysis and output of data lake data. It has the ability to handle various complex and changeable data query and visualization requirements, allowing enterprises and organizations to flexibly adjust analysis strategies and effectively respond to various challenges when facing different scenarios and problems. The use of large model reasoning technology allows non-professional users to easily perform complex data queries and visualization analysis without having a professional data processing background, thereby lowering the threshold for data analysis. It provides an intuitive and easy-to-use interface and interaction method, so that users can have a smooth and convenient experience during use, enhancing user satisfaction with data analysis tools.

[0127] In this embodiment, a data analysis device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0128] This embodiment provides a data analysis device, such as Figure 4 As shown, including:

[0129] A type determination module is used to obtain an original query question, classify the original query question, and determine the question type to which the original query question belongs;

[0130] A data entity identification module, used to identify the data entity in the original query question based on the question type, so as to determine the database table associated with the original query question according to the data entity;

[0131] A problem determination module, used to improve the data entity according to the metadata of the database table to determine the target query problem;

[0132] The result determination module is used to generate a target query statement based on the target query question to determine the data query result according to the target query statement.

[0133] In some optional implementations, the type determination module includes:

[0134] A vectorization processing unit, used for performing vectorization processing on the original query question;

[0135] The question type determination unit is used to perform logistic regression classification processing on the original query question after vectorization processing to determine the question type to which the original query question belongs.

[0136] In some optional implementations, the data entity identification module includes:

[0137] A model determination unit, used to determine a corresponding recognition model based on the problem type;

[0138] A data entity determination unit, configured to process the original query question based on the recognition model to determine the data entity;

[0139] A database query unit is used to query a database table associated with the original query question according to the data entity.

[0140] In some optional implementations, the problem determination module includes:

[0141] An adjustment judgment unit, used for judging whether the data entity needs to be adjusted based on a preset knowledge base and metadata of the database table;

[0142] The data entity improvement unit is used to improve the data entity if the data entity needs to be adjusted, so as to obtain a target query question.

[0143] In some optional implementations, the result determination module includes:

[0144] A target data determination unit, configured to filter the metadata of the database table based on a preset knowledge base and the target query question, and determine a target data field in the database table;

[0145] A statement generating unit, configured to generate a target query statement according to the preset knowledge base and the target query question;

[0146] A result generating unit is used to generate a data query result based on the target query statement and the target data field.

[0147] In some optional implementations, the result determination module includes:

[0148] A field conversion unit, used to convert a specific field in the target query question based on a preset knowledge base to determine a query field;

[0149] The second statement generating unit is used to generate a target query statement based on the query field.

[0150] In some optional embodiments, the device further comprises:

[0151] A resource acquisition module, used to acquire original data resources and pre-process the original data resources;

[0152] The knowledge base construction module is used to store the pre-processed original data resources into a database to obtain a preset knowledge base, which at least includes a vector database and a graph database.

[0153] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0154] The data analysis device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0155] The embodiment of the present invention also provides a computer device having the above Figure 4 The data analysis device shown.

[0156] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0157] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0158] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0159] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0161] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0162] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0163] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0164] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the present invention.

Claims

1. A data analysis method, characterized in that: The method comprises: Obtaining an original query question, and classifying the original query question to determine the question type to which the original query question belongs; Identify a data entity in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity; Completing the data entity according to the metadata of the database table to determine the target query question; A target query statement is generated based on the target query question to determine a data query result according to the target query statement.

2. The method according to claim 1, characterized in that The classifying the original query question to determine the question type to which the original query question belongs includes: Performing vectorization processing on the original query question; Logistic regression classification is performed on the original query question after vectorization processing to determine the question type to which the original query question belongs.

3. The method according to claim 1, characterized in that: The step of identifying a data entity in the original query question based on the question type, so as to determine a database table associated with the original query question according to the data entity, comprises: Determine a corresponding recognition model based on the problem type; Processing the original query question based on the recognition model to determine the data entity; A database table associated with the original query question is queried according to the data entity.

4. The method according to claim 1, characterized in that: The step of improving the data entity according to the metadata of the database table to determine the target query question includes: Determining whether the data entity needs to be adjusted based on a preset knowledge base and metadata of the database table; If the data entity needs to be adjusted, the data entity is improved to obtain the target query question.

5. The method according to claim 4, characterized in that Generating a target query statement based on the target query question to determine a data query result according to the target query statement includes: Filtering the metadata of the database table based on a preset knowledge base and the target query question to determine a target data field in the database table; Generate a target query statement according to the preset knowledge base and the target query question; A data query result is generated based on the target query statement and the target data field.

6. The method according to claim 5, characterized in that The generating a target query statement according to the preset knowledge base and the target query question includes: Transforming specific fields in the target query question based on a preset knowledge base to determine the query field; A target query statement is generated based on the query field.

7. The method according to claim 4, characterized in that The method further comprises: Acquire original data resources, and preprocess the original data resources; The preprocessed original data resources are stored in a database to obtain a preset knowledge base, which at least includes a vector database and a graph database.

8. A data analysis device, characterized in that: The device comprises: A type determination module is used to obtain an original query question, classify the original query question, and determine the question type to which the original query question belongs; A data entity identification module, used to identify the data entity in the original query question based on the question type, so as to determine the database table associated with the original query question according to the data entity; A problem determination module, used to improve the data entity according to the metadata of the database table to determine the target query problem; The result determination module is used to generate a target query statement based on the target query question to determine the data query result according to the target query statement.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data analysis method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data analysis method according to any one of claims 1 to 7.

Images