Data extraction method and device, equipment, storage medium and program product

Abstract

The invention discloses a data extraction method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a first demand description text input by a user, wherein the first demand description text is used for describing a data demand of the user; a data query statement is generated according to the first demand description text, the data query statement is expressed through a programming language, and the data query statement is used for extracting data from a data warehouse; converting the data query statement into a query statement description text, wherein the query statement description text is used for describing the content of the data query statement through a natural language; under the condition that first confirmation information corresponding to the query statement description text is obtained, data extraction is conducted through the data query statement, and the first confirmation information is used for indicating that the data query statement meets the data requirement of the user. According to the embodiment of the invention, the problem of low efficiency of data extraction in the prior art can be solved.

Description

Technical Field

[0001] The present application belongs to the field of information technology, and in particular, relates to a data extraction method, device, equipment, storage medium and program product. Background Art

[0002] With the advancement of big data informatization, the businesses of various enterprises have gathered a large amount of business data and stored it in data warehouses. In order to better utilize data resources, improve work efficiency and decision-making quality, and promote the sustainable development of business, enterprises usually have a large demand for data extraction.

[0003] In order to meet these data extraction needs, companies usually set up data management teams, store the business data of each department in a data warehouse, and let a dedicated team or person extract the required business data from the data warehouse by writing SQL statements. Since the writing of SQL statements is relatively complex, it will result in a longer data extraction time, so the data extraction efficiency is low. Summary of the invention

[0004] The embodiments of the present application provide a data extraction method, device, equipment, storage medium and program product, which can solve the problem of low efficiency of existing data extraction.

[0005] In a first aspect, an embodiment of the present application provides a method for extracting data, the method comprising:

[0006] Acquire a first requirement description text input by a user, where the first requirement description text is used to describe the data requirement of the user;

[0007] Generate a data query statement according to the first requirement description text, wherein the data query statement is expressed in a programming language and is used to extract data from a data warehouse;

[0008] Converting the data query statement into a query statement description text, wherein the query statement description text is used to describe the content of the data query statement in a natural language;

[0009] In the case of obtaining the first confirmation information corresponding to the query statement description text, the data query statement is used to extract data, and the first confirmation information is used to indicate that the data query statement meets the data needs of the user.

[0010] In some embodiments, generating a data query statement according to the first requirement description text includes:

[0011] Inputting the first requirement description text into a pre-trained first model to obtain a plurality of phrases output by the first model, wherein the plurality of phrases include the name of a first target model table and a first target field;

[0012] The multiple phrases are input into a pre-trained second model, and the second model generates the data query statement according to the multiple phrases, and the data query statement is used to extract the first target field from the first target model table in the data warehouse.

[0013] In some embodiments, inputting the plurality of phrases into a pre-trained second model comprises:

[0014] displaying the plurality of phrases;

[0015] When second confirmation information corresponding to the multiple phrases is received, the multiple phrases are input into a pre-trained second model, and the second confirmation information is used to confirm that the name of the first target model table and the first target field meet the user's data requirements.

[0016] In some embodiments, converting the data query statement into a query statement description text includes:

[0017] The feature vector corresponding to the data query statement is input into the pre-trained second model to obtain the query statement description text output by the second model.

[0018] In some embodiments, before inputting the first requirement description text into a pre-trained first model, the method further includes:

[0019] Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of the training samples includes a second requirement description text and a corresponding model table and a first prediction label of a field;

[0020] The first model is trained using the first training sample set and a pre-acquired database table until a training stop condition is met to obtain a trained first model, wherein the model library table is used to characterize the association relationship between multiple model tables, wherein the model library table is used to describe the association relationship between multiple model tables.

[0021] In some embodiments, before inputting the plurality of phrases into the pre-trained second model, the method further comprises:

[0022] Acquire a second training sample set, the training sample set comprising a plurality of second training samples, each of the second training samples comprising a third requirement description text and a second predicted label of a corresponding query statement description text;

[0023] The second model is trained using the second training sample set and the pre-acquired database table until a training stop condition is met, thereby obtaining a trained second model.

[0024] In some embodiments, the method further comprises:

[0025] Get metadata of model data from model tables in the data warehouse;

[0026] Performing a multi-dimensional analysis on the metadata according to a data analysis algorithm to obtain an analysis result, wherein the multi-dimensional analysis includes blood relationship analysis, correlation analysis, and impact analysis;

[0027] Establishing a data relationship between the metadata according to the analysis result;

[0028] A model library table is constructed according to the metadata and the data relationship.

[0029] In some embodiments, after generating a data query statement according to the first requirement description text, the method further includes:

[0030] Display the data query statement;

[0031] In response to the user's modification input to the data query statement, the data query statement is modified to obtain the modified data query statement.

[0032] In some embodiments, after converting the data query statement into a query statement description text, the method further includes:

[0033] When first rejection information corresponding to the query statement description text is obtained, an update prompt is generated, wherein the first rejection information is used to indicate that the data query statement does not meet the user's data requirements, and the update prompt is used to prompt to update the content of the data query statement.

[0034] In a second aspect, an embodiment of the present application provides a data extraction device, the device comprising:

[0035] An acquisition module, used to acquire a first requirement description text input by a user, where the first requirement description text is used to describe the data requirement of the user;

[0036] A generating module, configured to generate a data query statement according to the first requirement description text, wherein the data query statement is expressed in a programming language and is used to extract data from a data warehouse;

[0037] A conversion module, used to convert the data query statement into a query statement description text, wherein the query statement description text is used to describe the content of the data query statement in natural language;

[0038] The extraction module is used to extract data using the data query statement when first confirmation information corresponding to the query statement description text is obtained, and the first confirmation information is used to indicate that the data query statement meets the data needs of the user.

[0039] In a third aspect, an embodiment of the present application provides a data extraction device, the device comprising: a processor and a memory storing computer program instructions;

[0040] The above data extraction method is implemented when the processor executes the computer program instructions.

[0041] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and the computer program instructions implement the above data extraction method when executed by a processor.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the above data extraction method is implemented.

[0043] In the present application, through the data extraction method, device, equipment, storage medium and program product, the first demand description text input by the user can be obtained, and a data query statement can be generated according to the first demand description text. The data query statement is expressed by a programming language, and then the data query statement is converted into a query statement description text. When the first confirmation information corresponding to the query statement description text is obtained, the data query statement is used to extract data. In other words, the user only needs to put forward the first demand description text in natural language, and the natural language can be converted into a data query statement in a programming language, and then the data query statement is converted into a query statement description text described in natural language. As long as the user confirms the query statement description text, data can be extracted through the data query statement. In this way, the user can extract data by describing the data demand in natural language and confirm the query statement description text described in natural language, without the need for the user to write a programming language, thereby improving the efficiency of data extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of a method for extracting data provided by an embodiment of the present application;

[0046] Figure 2 is a flow chart of a method for extracting data provided by another embodiment of the present application;

[0047] Figure 3 is a flow chart of a method for extracting data provided by another embodiment of the present application;

[0048] Figure 4 It is a structural schematic diagram of a data extraction device provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of the hardware structure of a data extraction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0051] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0052] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0053] With the advancement of big data informatization, the businesses of various enterprises have gathered a large amount of business data and stored them in data warehouses. In order to better utilize data resources, improve work efficiency and decision-making quality, and promote the sustainable development of business, enterprises usually have a large number of data extraction needs. In order to meet these needs, enterprises usually establish a data management team, store a large amount of business data from various departments in the data warehouse, and let a dedicated team or personnel complete the corresponding data extraction needs. However, this will bring two problems. First, the operator needs to be familiar with the specific business of each department, but everyone's energy is limited, and the number of operators will increase with the increase of business; second, data extraction needs are usually completed through SQL (Structured Query Language), so the operator is generally a developer, and the time to write SQL is usually proportional to the difficulty of data extraction needs, so the efficiency of data extraction will become relatively low.

[0054] Specifically, in order to solve the problems of the prior art, the embodiments of the present application provide a data extraction method, device, equipment, storage medium and program product. The data extraction method provided by the embodiments of the present application is first introduced below.

[0055] Figure 1 A schematic diagram of a data extraction method provided by an embodiment of the present application is shown. The method comprises the following steps:

[0056] S110, obtaining a first requirement description text input by a user, where the first requirement description text is used to describe the data requirement of the user.

[0057] In this embodiment, a large amount of business data is stored in a pre-built data warehouse. Whenever a user needs to extract data from the data warehouse, a first requirement description text is input. The first requirement description text reflects the user's data requirement.

[0058] For example, the first requirement description text may include: the card number that needs to be assigned to Province A, the monthly traffic of the IoT card, the contract scenario of the IoT card, whether to activate traffic, whether to activate voice, whether to activate SMS, etc. For another example, the first requirement description text may also include: query the card number and customer name of the user in City B on January 30, 2024.

[0059] S120: Generate a data query statement according to the first requirement description text, wherein the data query statement is expressed in a programming language and is used to extract data from a data warehouse.

[0060] In this embodiment, the data query statement is a programming language, which is used to extract the required data from the data warehouse. The data query statement can be an SQL statement, an XPath statement, or an XQuery statement. After obtaining the first requirement description text input by the user, the first requirement description text input by the user can be parsed. Then, key elements such as model tables, fields, conditions, and sorting methods in the first requirement description text are extracted. Then, a corresponding data query statement can be constructed based on the extracted key elements.

[0061] As an optional embodiment, the above S120 may include:

[0062] Inputting the first requirement description text into a pre-trained first model to obtain a plurality of phrases output by the first model, wherein the plurality of phrases include the name of a first target model table and a first target field;

[0063] The multiple phrases are input into a pre-trained second model, and the second model generates the data query statement according to the multiple phrases, and the data query statement is used to extract the first target field from the first target model table in the data warehouse.

[0064] In this embodiment, the model table is a table structure commonly used in database design and data analysis, which is used to describe and store specific data. The data warehouse includes multiple model tables, each of which stores different business data. The first target model table is the model table where the business data corresponding to the first requirement description text is located, and the first target field is the business data corresponding to the first requirement description text in the first target model table.

[0065] After receiving the first requirement description text input by the user, multiple phrases can be extracted from the first requirement description text by the first model, and the multiple phrases can be input into the second model, and the second model can convert the multiple phrases into data extraction statements.

[0066] For example, the first model can be a Seq2seq model based on hybrid attention, which is an improved sequence-to-sequence (Seq2seq) model, wherein the first model consists of an encoder and a decoder. The encoder is used to encode the input sequence into a fixed-length context vector to capture the semantic information of the input sequence. The decoder is used to generate a target sequence based on the context vector.

[0067] By way of example, the second model may be a ChatGLM (Chat General Language Model) model.

[0068] As an optional embodiment, inputting the multiple phrases into a pre-trained second model includes:

[0069] displaying the plurality of phrases;

[0070] Upon receiving second confirmation information corresponding to the plurality of phrases, the plurality of phrases are input into a pre-trained second model, wherein the second confirmation information is used to confirm that the name of the first target model table and the first target field match the user's data requirements.

[0071] The multiple phrases include key elements such as the first target model table and the first target field described in the first requirement description text. Subsequently, the multiple phrases can be displayed, and the user confirms whether the first target model table and the first target field match the user's data requirements. If the user believes that the multiple phrases match the user's data requirements, the user enters second confirmation information. After receiving the second confirmation information, the multiple phrases can be entered into the second model, and the second model generates data extraction statements based on the multiple phrases.

[0072] In addition, if the user believes that the multiple phrases do not match the user's data requirements, the user enters second rejection information. After receiving the second rejection information, the electronic device modifies the first requirement description text and generates multiple phrases based on the modified first requirement description text.

[0073] In this way, it is possible to ensure that key elements such as the first target model table and the first target field are accurately extracted from the first requirement description text, thereby ensuring the accuracy of data extraction.

[0074] S130: Convert the data query statement into a query statement description text, where the query statement description text is used to describe the content of the data query statement in a natural language.

[0075] In this embodiment, after obtaining the data query statement generated according to the first requirement description text, the generated data query statement can be parsed, and then the key elements in the data query statement can be identified, and a natural language description text, i.e., a query statement description text, can also be generated based on the key elements.

[0076] As an optional embodiment, converting the data query statement into a query statement description text includes:

[0077] The feature vector corresponding to the data query statement is input into the pre-trained second model to obtain the query statement description text output by the second model.

[0078] In this embodiment, after the multiple phrases are input into the second model, the second model may convert the multiple phrases into data extraction statements, then convert the data extraction statements into corresponding feature vectors, and output the feature vectors corresponding to the data query statements.

[0079] Subsequently, the feature vector corresponding to the data query statement can be further input into the second model, and the second model can convert the feature vector corresponding to the data query statement into a query statement description text expressed in a natural language.

[0080] S140, when first confirmation information corresponding to the query statement description text is obtained, data extraction is performed using the data query statement, wherein the first confirmation information is used to indicate that the data query statement meets the data requirements of the user.

[0081] In this embodiment, the first confirmation information is used to indicate that the data query statement meets the user's data needs. After generating a query statement description text describing the data query statement, the query statement description text can be displayed to allow the user to determine whether the data query statement meets the user's data needs by reading the query statement description text. If the user believes that the data query statement meets his or her data needs, the first confirmation information can be entered. After the electronic device receives the first confirmation information, the data query statement can be entered into the corresponding data extraction script, and the corresponding data can be extracted from the data warehouse through the data extraction script.

[0082] In the present application, through the data extraction method, the first demand description text input by the user can be obtained, and a data query statement can be generated according to the first demand description text. The data query statement is expressed by a programming language, and then the data query statement is converted into a query statement description text. When the first confirmation information corresponding to the query statement description text is obtained, the data query statement is used to extract data. In other words, the user only needs to put forward the first demand description text in natural language, and then the natural language can be converted into a data query statement in a programming language, and then the data query statement is converted into a query statement description text described in natural language. As long as the user confirms the query statement description text, data can be extracted through the data query statement. In this way, the user can extract data by describing the data demand in natural language and confirm the query statement description text described in natural language, without the need for the user to write a programming language, thereby improving the efficiency of data extraction.

[0083] As an optional embodiment, after converting the data query statement into a query statement description text, the method further includes:

[0084] When first rejection information corresponding to the query statement description text is obtained, an update prompt is generated, wherein the first rejection information is used to indicate that the data query statement does not meet the user's data requirements, and the update prompt is used to prompt to update the content of the data query statement.

[0085] In this embodiment, similarly, after generating a query statement description text describing the data query statement, the query statement description text can be displayed to allow the user to determine whether the data query statement meets the user's data needs by reading the query statement description text. If the user believes that the data query statement does not meet his or her data needs, a first rejection message can be input. After the electronic device receives the first rejection message, a corresponding update prompt can be output to prompt the update of the data query statement. The data query statement can be adjusted to meet the user's data needs.

[0086] Through the above implementation, before data extraction, the user can determine whether the data query statement can meet his needs. If the content of the data query statement does not meet his data needs, the data query statement needs to be adjusted and updated.

[0087] As an optional embodiment, before inputting the first requirement description text into the pre-trained first model, the method further includes:

[0088] Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of the training samples includes a second requirement description text and a corresponding model table and a first prediction label of a field;

[0089] The first model is trained using the first training sample set and the pre-acquired database table until a training stop condition is met, thereby obtaining a trained first model, wherein the model database table is used to characterize the association relationship between the plurality of model tables.

[0090] In this embodiment, the second requirement description text is used to describe the user's data requirements. Taking the first model as a Seq2seq model based on hybrid attention as an example, a Bi-LSTM (Bidirectional Long Short-Term Memory) is used to obtain the model table name and other related information of the first prediction label as global information.

[0091] like Figure 2As shown, the given input of the first model consists of multiple words (W1, W2, ..., WL), including the second demand description text, the first prediction label and the key information in the database table. Then, the multiple words (W1, W2, ..., WL) are processed into vector representations (X1, X2, ..., XL) through the embedding layer. Then, the bidirectional long short-term memory network (Bidirectional LSTM) in the first model can be used to process the input sequence (X1, X2, ..., XL), encode from front to back and from back to front, and generate global information hG. The global information hG can be specifically expressed as:

[0092]

[0093] Among them, the encoding formula can be expressed as:

[0094]

[0095] Among them, Xi is the word embedding vector of the current time step, is the hidden state at the current time step, is the hidden state at the next time step.

[0096] After obtaining the global information hG, the multi-head attention mechanism can be used to combine the global information hG obtained from Bi-LSTM and the local information hL generated by the attention mechanism to provide rich contextual information. Among them, the local information hL can be obtained using the multi-head attention mechanism (MHA). The attention mechanism is used to expand the model's ability to focus on multiple positions in the sequence and refine the representation of the model table description in the interactive information. The matrix uses the weights of the hidden layer obtained after the GLOVE model training is completed as the vector representation of the word:

[0097]

[0098] Where X is the input sequence represented by the vector, Wq is the weight matrix used to generate the query vector (Query), Wk is the weight matrix used to generate the key vector, Wv is the weight matrix used to generate the value vector, and dk is the dimension of the key vector. Headi represents the single-head attention output of the i-th head. n represents the number of heads. W0 is the linear transformation matrix after the multi-head attention output. Represents local information, that is, the local correlation semantic information of the input sequence calculated by the multi-head attention mechanism.

[0099] At the same time, the first model also includes a label embedding layer module, which can implicitly fuse the first predicted label in the first training sample set using the calculation method of the word vector average function as the input of the decoder. The decoder gradually generates an output sequence (Y1, Y2, Y3, Y4). Each decoding step uses global information hG, local information hL and context vectors. The decoder is composed of LSTM units. During the use of the decoder, the label embedding layer (LabelEmbedding, LE) is used to convert the first predicted label into an embedded vector, and the mask vector can be used to avoid repetition of the first predicted label.

[0100] In this way, it can be determined whether the output sequence (Y1, Y2, Y3, Y4) meets the training stop condition. If the training stop condition is met, the training is stopped and the trained first model is obtained. Otherwise, the training is continued until the training stop condition is met. In the above manner, the trained first model can be obtained.

[0101] As an optional embodiment, before inputting the plurality of phrases into the pre-trained second model, the method further includes:

[0102] Acquire a second training sample set, the training sample set comprising a plurality of second training samples, each of the second training samples comprising a third requirement description text and a second predicted label of a corresponding query statement description text;

[0103] The second model is trained using the second training sample set and the pre-acquired database table until a training stop condition is met, thereby obtaining a trained second model.

[0104] In this embodiment, taking the second model as the ChatGLM model as an example, the ChatGLM model is used to convert SQL statements into natural language and vice versa, wherein the third requirement description text is used to describe the user's data requirements, wherein the SQL statement is a data query statement.

[0105] During the training process of the second model, the second training sample set can be obtained, and then the second training samples in the second training sample set are divided into a training set and a test set at a ratio of 8:2. The ChatGLM model is used to segment and vectorize the third demand description text in the training set, and then the segmentation of the vectorized third demand description text is input into the ChatGLM model. The ChatGLM model converts the vectorized segmentation into SQL statements and outputs the vector features corresponding to the SQL statements. Subsequently, the vector features corresponding to the SQL statements can be input into the second model to obtain the query statement description text output by the second model. The query statement description text output by the second model and the second predicted label can be evaluated until the query statement description text output by the second model matches the second predicted label, and it is considered that the second model meets the training stop condition, and the trained second model is obtained.

[0106] Through the above method, a trained second model can be obtained.

[0107] As an optional embodiment, before training the first model using the first training sample set and the pre-acquired database table, the method further includes:

[0108] Get metadata of model data from model tables in the data warehouse;

[0109] Performing a multi-dimensional analysis on the metadata according to a data analysis algorithm to obtain an analysis result, wherein the multi-dimensional analysis includes blood relationship analysis, correlation analysis, and impact analysis;

[0110] Establishing a data relationship between the metadata according to the analysis result;

[0111] A model library table is constructed according to the metadata and the data relationship.

[0112] In this embodiment, if Figure 3 As shown, the metadata in the data warehouse can be firstly subjected to data stratification and dimensional modeling. Specifically, metadata is extracted from the model table in the data warehouse through data governance, and lineage analysis, correlation analysis and impact analysis are performed on the metadata, and the analysis results are used as external knowledge for hybrid model processing.

[0113] Among them, through metadata lineage analysis, we can understand where the metadata comes from and the means of processing; through metadata correlation analysis, we can sort out how the relationship between model tables is established, and we can understand the other entities associated with a certain entity and the processing processes it participates in to view the usage of specific data from two perspectives, forming a network of entities and the processing processes they participate in; through metadata impact analysis, we can infer the association relationship of the data, trace it downward, and quickly find out which applications or data warehouse layers use this data.

[0114] Therefore, this proposal uses metadata and the analysis results of metadata to sort out the models of the model tables in the data warehouse and establish a model library table. The model library table is mainly composed of six fields in the data warehouse, namely, the model table ID, model table name, model table description, blood relationship, association relationship and influence relationship. These six fields can describe the meaning of the model table itself and the association relationship between a model table and other model tables.

[0115] The model library table is shown in Table 1 below:

[0116] Table 1 Model library table

[0117]

[0118] Through the above method, the model table of the data warehouse can be extracted to present the model table name, model table description, model table blood relationship, association relationship and influence relationship, so as to assist the first model and the second model to accurately select the model table involved in the data demand.

[0119] As an optional embodiment, after generating a data query statement according to the first requirement description text, the method further includes:

[0120] Display the data query statement;

[0121] In response to the user's modification input to the data query statement, the data query statement is modified to obtain the modified data query statement.

[0122] In this embodiment, after generating a data query statement based on the first requirement description text, in order to avoid errors in the automatically generated data query statement for certain specific scenarios, the data query statement can be displayed on the display interface of the electronic device. If the user believes that the data query statement needs to be corrected, the data query statement can be modified and input to modify the content of the data query statement to avoid errors in the data query statement affecting data extraction requirements.

[0123] In addition, for more difficult or complex data extraction requirements, developers can use custom functions, time functions, row-to-column conversion, conditional statements, deduplication, concat and other methods to modify the model table, and can also dynamically adjust the model table and the dependencies between model tables according to the usage scenario. After completing the above adjustments, this proposal will execute the data query statement based on the modified data query statement and output multiple execution results. Developers can view the execution results and dynamically adjust the model table and the dependencies between model tables based on the execution results.

[0124] Through the above method, when there are errors in the scene, it is possible to avoid errors in data query statements caused by the scene being too complicated, thereby ensuring the accuracy of the extracted data.

[0125] In addition, the present application requires repeated confirmation from the user during text preprocessing and SQL statement generation and verification, thereby ensuring the consistency of requirements and the accuracy of generated results, and reducing the possibility of subsequent errors.

[0126] Based on the data extraction method provided in the above embodiment, the present application also provides a specific implementation of a data extraction device. Please refer to the following embodiment.

[0127] See first Figure 4 The data extraction device 400 provided in the embodiment of the present application includes the following modules:

[0128] An acquisition module 401 is used to acquire a first requirement description text input by a user, where the first requirement description text is used to describe the data requirement of the user;

[0129] A generating module 402, configured to generate a data query statement according to the first requirement description text, wherein the data query statement is expressed in a programming language and is used to extract data from a data warehouse;

[0130] A conversion module 403, used to convert the data query statement into a query statement description text, wherein the query statement description text is used to describe the content of the data query statement in natural language;

[0131] The extraction module 404 is used to extract data using the data query statement when first confirmation information corresponding to the query statement description text is obtained, and the first confirmation information is used to indicate that the data query statement meets the data requirements of the user.

[0132] The device can obtain the first demand description text input by the user, generate a data query statement based on the first demand description text, the data query statement is expressed in a programming language, and then the data query statement is converted into a query statement description text. When the first confirmation information corresponding to the query statement description text is obtained, the data query statement is used to extract data. In other words, the user only needs to put forward the first demand description text in natural language, and then the natural language can be converted into a data query statement in a programming language, and then the data query statement is converted into a query statement description text described in natural language. As long as the user confirms the query statement description text, data can be extracted through the data query statement. In this way, the user can extract data by describing the data demand in natural language and confirm the query statement description text described in natural language without the user having to write a programming language, thereby improving the efficiency of data extraction.

[0133] As an implementation of the present application, the generation module 402 may also be used to:

[0134] Inputting the first requirement description text into a pre-trained first model to obtain a plurality of phrases output by the first model, wherein the plurality of phrases include the name of a first target model table and a first target field;

[0135] The multiple phrases are input into a pre-trained second model, and the second model generates the data query statement according to the multiple phrases, and the data query statement is used to extract the first target field from the first target model table in the data warehouse.

[0136] As an implementation of the present application, the generation module 402 may also be used to:

[0137] displaying the plurality of phrases;

[0138] When second confirmation information corresponding to the multiple phrases is received, the multiple phrases are input into a pre-trained second model, and the second confirmation information is used to confirm that the name of the first target model table and the first target field meet the user's data requirements.

[0139] As an implementation of the present application, the conversion module 403 may also be used to:

[0140] The feature vector corresponding to the data query statement is input into the pre-trained second model to obtain the query statement description text output by the second model.

[0141] As an implementation of the present application, the data extraction device 400 may also be used for:

[0142] Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of the training samples includes a second requirement description text and a corresponding model table and a first prediction label of a field;

[0143] The first model is trained using the first training sample set and a pre-acquired database table until a training stop condition is met to obtain a trained first model, wherein the model library table is used to characterize the association relationship between multiple model tables, wherein the model library table is used to describe the association relationship between multiple model tables.

[0144] As an implementation of the present application, the data extraction device 400 may also be used to: obtain a second training sample set, the training sample set comprising a plurality of second training samples, each of the second training samples comprising a third requirement description text and a second predicted label of its corresponding query statement description text;

[0145] The second model is trained using the second training sample set and the pre-acquired database table until a training stop condition is met, thereby obtaining a trained second model.

[0146] As an implementation of the present application, the data extraction device 400 may also be used for:

[0147] Get metadata of model data from model tables in the data warehouse;

[0148] Performing a multi-dimensional analysis on the metadata according to a data analysis algorithm to obtain an analysis result, wherein the multi-dimensional analysis includes blood relationship analysis, correlation analysis, and impact analysis;

[0149] Establishing a data relationship between the metadata according to the analysis result;

[0150] A model library table is constructed according to the metadata and the data relationship.

[0151] As an implementation of the present application, the data extraction device 400 may also be used for:

[0152] Display the data query statement;

[0153] In response to the user's modification input to the data query statement, the data query statement is modified to obtain the modified data query statement.

[0154] As an implementation of the present application, the data extraction device 400 may also be used for:

[0155] When first rejection information corresponding to the query statement description text is obtained, an update prompt is generated, wherein the first rejection information is used to indicate that the data query statement does not meet the user's data requirements, and the update prompt is used to prompt to update the content of the data query statement.

[0156] The data extraction device provided in the embodiment of the present invention can implement each step in the above method embodiment, and will not be described again here to avoid repetition.

[0157] Figure 5 A schematic diagram of the hardware structure of a data extraction device provided in an embodiment of the present application is shown.

[0158] The data extraction device may include a processor 1001 and a memory 1002 storing computer program instructions.

[0159] Specifically, the processor 1001 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0160] The memory 1002 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1002 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 1002 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 1002 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 1002 is a non-volatile solid-state memory.

[0161] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0162] The processor 1001 implements any one of the data extraction methods in the above embodiments by reading and executing computer program instructions stored in the memory 1002 .

[0163] In one example, the data extraction device may further include a communication interface 1003 and a bus 1010. Figure 5 As shown, the processor 1001, the memory 1002, and the communication interface 1003 are connected via a bus 1010 and communicate with each other.

[0164] The communication interface 1003 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0165] Bus 1010 includes hardware, software or both, and the parts of the extraction equipment of data are coupled to each other.For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, bus 1010 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0166] The data extraction device may be based on the above-mentioned embodiment, thereby realizing the data extraction method and apparatus combined with the above-mentioned embodiment.

[0167] In addition, in combination with the data extraction method in the above-mentioned embodiment, the embodiment of the present application may provide a computer storage medium to implement. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the data extraction methods in the above-mentioned embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the above-mentioned computer-readable storage medium may include a non-transitory computer-readable storage medium, such as a read-only memory (Read-Only Memory, referred to as ROM), a random access memory (Random Access Memory, referred to as RAM), a disk or an optical disk, etc., which is not limited here.

[0168] In addition, an embodiment of the present application further provides a computer program product, including computer program instructions, which can implement the steps and corresponding contents of the aforementioned method embodiment when the computer program instructions are executed by a processor.

[0169] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0170] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0171] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0172] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0173] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for extracting data, characterized in that: The method comprises: Acquire a first requirement description text input by a user, where the first requirement description text is used to describe the data requirement of the user; Generate a data query statement according to the first requirement description text, wherein the data query statement is expressed in a programming language and is used to extract data from a data warehouse; Converting the data query statement into a query statement description text, wherein the query statement description text is used to describe the content of the data query statement in a natural language; In the case of obtaining the first confirmation information corresponding to the query statement description text, the data query statement is used to extract data, and the first confirmation information is used to indicate that the data query statement meets the data needs of the user.

2. The data extraction method according to claim 1, characterized in that: The step of generating a data query statement according to the first requirement description text includes: Inputting the first requirement description text into a pre-trained first model to obtain a plurality of phrases output by the first model, wherein the plurality of phrases include the name of a first target model table and a first target field; The multiple phrases are input into a pre-trained second model, and the second model generates the data query statement according to the multiple phrases, and the data query statement is used to extract the first target field from the first target model table in the data warehouse.

3. The data extraction method according to claim 2, characterized in that: The step of inputting the plurality of phrases into a pre-trained second model comprises: displaying the plurality of phrases; When second confirmation information corresponding to the multiple phrases is received, the multiple phrases are input into a pre-trained second model, and the second confirmation information is used to confirm that the name of the first target model table and the first target field meet the user's data requirements.

4. The data extraction method according to claim 1, characterized in that: The step of converting the data query statement into a query statement description text comprises: The feature vector corresponding to the data query statement is input into the pre-trained second model to obtain the query statement description text output by the second model.

5. The data extraction method according to claim 2, characterized in that: Before inputting the first requirement description text into the pre-trained first model, the method further includes: Acquire a first training sample set, wherein the first training sample set includes a plurality of first training samples, each of the training samples includes a second requirement description text and a corresponding model table and a first prediction label of a field; The first model is trained using the first training sample set and a pre-acquired database table until a training stop condition is met to obtain a trained first model, wherein the model library table is used to characterize the association relationship between multiple model tables, wherein the model library table is used to describe the association relationship between multiple model tables.

6. The data extraction method according to claim 2, characterized in that: Before inputting the plurality of phrases into the pre-trained second model, the method further comprises: Acquire a second training sample set, the training sample set comprising a plurality of second training samples, each of the second training samples comprising a third requirement description text and a second predicted label of a corresponding query statement description text; The second model is trained using the second training sample set and the pre-acquired database table until a training stop condition is met, thereby obtaining a trained second model.

7. The data extraction method according to claim 5 or 6, characterized in that: The method further comprises: Get metadata of model data from model tables in the data warehouse; Performing a multi-dimensional analysis on the metadata according to a data analysis algorithm to obtain an analysis result, wherein the multi-dimensional analysis includes blood relationship analysis, correlation analysis, and impact analysis; Establishing a data relationship between the metadata according to the analysis result; A model library table is constructed according to the metadata and the data relationship.

8. The data extraction method according to claim 1, characterized in that: After generating a data query statement according to the first requirement description text, the method further includes: Display the data query statement; In response to the user's modification input to the data query statement, the data query statement is modified to obtain the modified data query statement.

9. The data extraction method according to claim 1, characterized in that: After converting the data query statement into a query statement description text, the method further includes: When first rejection information corresponding to the query statement description text is obtained, an update prompt is generated, wherein the first rejection information is used to indicate that the data query statement does not meet the user's data requirements, and the update prompt is used to prompt to update the content of the data query statement.

10. A data extraction device, characterized in that: The device comprises: An acquisition module, used to acquire a first requirement description text input by a user, where the first requirement description text is used to describe the data requirement of the user; A generating module, configured to generate a data query statement according to the first requirement description text, wherein the data query statement is expressed in a programming language and is used to extract data from a data warehouse; A conversion module, used to convert the data query statement into a query statement description text, wherein the query statement description text is used to describe the content of the data query statement in natural language; The extraction module is used to extract data using the data query statement when first confirmation information corresponding to the query statement description text is obtained, and the first confirmation information is used to indicate that the data query statement meets the data needs of the user.

11. A data extraction device, characterized in that: The data extraction device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data extraction method according to any one of claims 1 to 9 is implemented.

12. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, which, when executed by a processor, implement the data extraction method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises computer program instructions, and when the computer program instructions are executed by a processor, the data extraction method according to any one of claims 1 to 9 is implemented.

Images