A data processing method, device, equipment, and storage medium

Abstract

The present application discloses a data processing method, apparatus, device and storage medium, the method comprising: obtaining query content; determining at least one query field, an identifier of a target data model and at least one output field based on the query content and a metadata database; the metadata database stores fields in at least two data models; using a running data query engine, obtaining query results based at least on the at least one query field, the identifier of the target data model and the at least one output field. The solution can remove the limitation of business scenarios and improve the query accuracy in complex business scenarios.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and relates to but is not limited to a data processing method and apparatus, equipment, and storage medium. Background Art

[0002] With the development of artificial intelligence (AI) and natural language processing (NLP) technologies, intelligent question-answering products (for example, Apple's voice assistant, Microsoft XiaoIce, and JD's intelligent customer service) are becoming more and more widely used in e-commerce, retail, finance, transportation and other industries, as well as in daily life.

[0003] At the same time, with the development of business intelligence (BI), intelligent question and answer for structured databases has attracted the attention of more and more companies and researchers. Relevant research from natural language to structured query language has been widely developed. Intelligent question and answer products can use pre-trained deep learning and other models to process the natural language input by users through semantic analysis and give intuitive answers.

[0004] However, the intelligent question-and-answer products in the related technologies can only query the answers to questions in a single data model for a single business scenario; when faced with complex business scenarios, querying answers in a single business scenario cannot guarantee the accuracy of the query results. Summary of the invention

[0005] The present application provides a data processing method and apparatus, equipment, and storage medium, which can remove the limitations of business scenarios and improve query accuracy in complex business scenarios.

[0006] The technical solution of this application is implemented as follows:

[0007] The present application provides a data processing method, the method comprising:

[0008] Get the query content;

[0009] Determining at least one query field, an identifier of a target data model and at least one output field based on the query content and a metadata database storing fields in at least two data models;

[0010] By using the running data query engine, a query result is obtained based on at least the at least one query field, the identifier of the target data model and the at least one output field.

[0011] The present application provides a data processing device, the device comprising:

[0012] An acquisition unit, used for acquiring query content;

[0013] a determination unit, configured to determine at least one query field, an identifier of a target data model, and at least one output field based on the query content and a metadata database storing fields in at least two data models;

[0014] The processing unit is used to obtain a query result based on at least the at least one query field, the identifier of the target data model and the at least one output field by using the running data query engine.

[0015] The present application also provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned data processing method when executing the program.

[0016] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0017] The data processing method, device, equipment and storage medium provided by the present application include obtaining query content; determining at least one query field, an identifier of a target data model and at least one output field based on the query content and a metadata database; the metadata database stores fields in at least two data models; using a running data query engine, obtaining query results based at least on the at least one query field, the identifier of the target data model and the at least one output field. In this solution, the fields included in multiple data models are integrated into the metadata database, so that by querying the metadata database, it is possible to find answers to questions in multiple data models, with high query accuracy; and the limitation on business scenarios can be removed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An optional structural diagram of a data processing system provided in an embodiment of the present application;

[0019] Figure 2 An optional structural diagram of a data processing terminal provided in an embodiment of the present application;

[0020] Figure 3 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0021] Figure 4 An optional flowchart of a data processing method provided in an embodiment of the present application;

[0022] Figure 5 An optional flow chart of the data processing method provided in the embodiment of the present application;

[0023] Figure 6 An optional flow chart of the data processing method provided in the embodiment of the present application;

[0024] Figure 7 An optional flow chart of the data processing method provided in the embodiment of the present application;

[0025] Figure 8 An optional flow chart of the data processing method provided in the embodiment of the present application;

[0026] Fig. 9 An optional structural diagram of the multi-table intelligent question-answering system provided in the embodiment of the present application;

[0027] Fig.10 An optional flow chart of the data processing method provided in the embodiment of the present application;

[0028] Fig.11 A schematic diagram of an optional structure of a text input unit provided in an embodiment of the present application;

[0029] Fig.12 An optional structural diagram of a data processing device provided in an embodiment of the present application;

[0030] Fig.13 An optional structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" are used only as examples to distinguish different objects, and do not represent a specific order for the objects, nor do they have a limitation on the order of precedence. It is understandable that "first\second\third" can be interchanged with a specific order or order of precedence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0035] In order to facilitate understanding of the technical solution of the present application, the nouns or technical terms involved in the present application are explained below.

[0036] 1) Data model refers to a model that includes multiple items and the values ​​of each item.

[0037] In one example, the data model may be in the form of a two-dimensional table; Table 1 illustrates a data model A, which includes the grades of students in various subjects; Table 2 illustrates a data model B, which includes basic information of students.

[0038] Table 1. Data model example

[0039] Student Name Language scores Mathematics scores English scores Sports performance Zhang San 85 98 71 96 Li Si 95 94 88 91 …… …… …… …… …… Wang Wu 90 88 98 86

[0040] Table 2. Data model example

[0041] Student Name gender Height (cm) Home address Health status Zhang San male 155 Address A good Li Si female 156 Address B good …… …… …… …… Wang Wu male 158 Address C good

[0042] In another example, the data model may also be in a graphical form, an object notation (JavaScript Object Notation, json) form, a dictionary form, and the like.

[0043] 2) Field refers to an item in the data model. In one example, a field can refer to the name of a column in a two-dimensional table; for example, a field can be a student's name in Table 1; another example is a Chinese language score in Table 1.

[0044] 3) Query fields refer to the fields used for querying in the data model that are determined by the query content. For example, if the query content is "What is the average score for Chinese?", then the query field can be considered to be the Chinese score.

[0045] 4) Output field refers to the field determined by the query content and used to represent the query results. Corresponding to Example 1, the output field is the Chinese language score in Table 1.

[0046] The embodiments of the present application may provide a data processing method and apparatus, a device and a storage medium. In practical applications, the data processing method may be implemented by a data processing apparatus, and each functional entity in the data processing apparatus may be collaboratively implemented by hardware resources of an electronic device (such as a terminal device), such as computing resources such as a processor, and communication resources (such as for supporting various communication methods such as optical cables and cellular communications).

[0047] The data processing method provided in the embodiment of the present application is applied to a data processing system, and the data processing system includes a data processing terminal. In one example, the data processing system may also include a client.

[0048] As an example, the structure of the data processing system can be as follows Figure 1 As shown, it includes: a client 10 and a data processing terminal 20.

[0049] In one example, the client 10 and the data processing end 20 may be the same physical entity; in another example, Figure 1 As shown, the client 10 and the data processing end 20 may be different physical entities, and the client 10 and the data processing end 20 interact with each other via a network 30 .

[0050] Here, the client 10 is used to receive user operations, and based on the user operations, sends query content to the data processing end 20. The data processing end 20 is used to receive the query content sent by the client 10, and obtain answers to the query content in at least two data models according to the query content, and output them.

[0051] In one example, the structure of the data processing end can be as follows Figure 2 As shown, the data processing end 20 includes: a semantic analysis module 201, a data storage module 202 and a background processing module 203.

[0052] Among them, the semantic parsing module 201 is used to receive and process the data sent by the client 10. For example, in combination with the embodiments of the present application, the semantic parsing module 201 can be used to receive the query content sent by the client 10, and parse the query content to obtain the structured query conditions corresponding to the query content (including at least: at least one query field, the identifier of the target data model and at least one output field); the semantic parsing module 201 can also be used to send the structured query conditions to the client so that the structured query conditions can be displayed on the client; the semantic parsing module 201 can also be used to sort the fields in multiple data models, and send the fields included in the multiple models and the identifiers of the data models to which the fields belong to, and other information to the data storage module 202.

[0053] The data storage module 202 is used to store data. For example, in combination with the embodiments of the present application, the data storage module 202 may include a metadata library, which is used to store information such as the fields included in the multiple models sent by the semantic parsing module 201 and the identifiers of the data models to which the fields belong; the data storage module 202 may also include a Frequently Asked Questions (FAQ) sub-library, and the FAQ library is used to store frequently asked questions (also referred to as query content or question content) and structured query conditions corresponding to the frequently asked questions. It is understandable that the FAQ library may also be stored as a sub-library in the metadata library.

[0054] The background processing module 203 is used to process relevant data. For example, in combination with the embodiments of the present application, the background processing module 203 can be used to determine the answer (also referred to as the query result) corresponding to the query content in multiple data models based on the structured query conditions; the background processing module 203 can also be used to organize the answer corresponding to the query content into a data report or a visual graphic format, and send it to the client 10 so that the client can display the answer organized into a data report or a visual graphic format; the background processing module 203 can also be used to search for the structured query conditions corresponding to the FAQ library.

[0055] In the present application embodiment, based on Figure 1 In the data processing system shown, the client sends query content to the data processing end, the data processing end receives the query content, and executes: determining at least one query field, an identifier of a target data model, and at least one output field based on the query content and a metadata database; the metadata database stores fields in at least two data models; and using a running data query engine, obtaining a query result based at least on the at least one query field, the identifier of the target data model, and the at least one output field.

[0056] Next, combine Figure 1 The schematic diagram of the data processing system shown illustrates various embodiments of the data processing method and apparatus, device and storage medium provided in the embodiments of the present application.

[0057] This embodiment provides a data processing method, which is applied to a data processing device, wherein the data processing device can be implemented on an electronic device as a data processing end. The function implemented by the method can be implemented by a processor in the electronic device calling a program code, and of course the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0058] The electronic device may be any device capable of processing relevant information. In one embodiment, the electronic device may be a server.

[0059] Of course, the embodiments of the present application are not limited to the provided methods and hardware, and there may be a variety of implementation methods, such as providing a storage medium (storing instructions for executing the data processing method provided by the embodiments of the present application).

[0060] The data processing method provided in the embodiment of the present application is described below.

[0061] Figure 3 The flowchart of a data processing method of an embodiment of the present application is used to determine the query result of the query content. The processing process for each query content is similar. Now, taking the query content as an example, the process of how to obtain the query result corresponding to the query content is described in detail.

[0062] The data processing method may include but is not limited to Figure 3 The following S301 to S303 are shown.

[0063] S301. The electronic device obtains query content.

[0064] S301 may be implemented as follows: the electronic device receives data content input by a client or sent by other devices through a data transmission interface, and uses the data content as query content, or converts the data content into query content.

[0065] The embodiment of the present application does not specifically limit the form of obtaining the query content, and can be configured according to actual needs. For example, the query content can be received by the electronic device in the form of text through the text input interface on the client device; for another example, the query content can be received by the electronic device in the form of voice through the voice input interface on the client device, and then the voice is converted into text; for another example, the query content can be received by the electronic device through the data transmission interface between other devices or converted after receiving.

[0066] The specific access form of the client can be configured according to actual needs, and the embodiment of the present application does not specifically limit this. For example, the access form of the client can be in the form of a World Wide Web (web) web page or an application (Application, App).

[0067] For example, S301 may be implemented as follows: the electronic device receives the input of "the number of people who have resigned from Department A in the past three months" through a text input interface on a web page, and uses "the number of people who have resigned from Department A in the past three months" as query content.

[0068] S302: The electronic device determines at least one query field, an identifier of a target data model, and at least one output field based on the query content and the metadata database.

[0069] The metadata library stores fields in at least two data models; different data models store data contents of different business scenarios.

[0070] In one example, the metadata database further stores the identifier of the data model to which the field belongs; wherein the identifier of the data model uniquely points to a data model.

[0071] The embodiment of the present application does not specifically limit the content stored in the metadata library and can be configured according to actual needs.

[0072] S302 may be implemented as follows: the electronic device performs content parsing on the query content, and determines at least one query field, an identifier of a target data model, and at least one output field according to the content of the parsed query content and the content in the metadata database.

[0073] It is understandable that at least one query field, an identifier of a target data model, and at least one output field may also be packaged into a structured query condition.

[0074] The embodiment of the present application does not specifically limit the format of the structured query condition, as long as it can be recognized by the data query engine. For example, the structured query condition can be a structured query language (SQL) statement.

[0075] Example A, the query content is "What are the Chinese scores of each student?", the metadata database may include: (student name, data model A and data model B), (Chinese score, data model A), (math score, data model A), (English score, data model A), (physical education score, data model A), (gender, data model B), (height, data model B), (home address, data model B), (health status, data model B); the content in brackets includes a field and the identifier of the data model to which the word belongs; then the electronic device determines at least one query field, the identifier of the target data model and at least one output field after S302 include: the query field includes the student's name and Chinese score, the target data model includes data model A, and the output field includes the student's name and Chinese score.

[0076] S303: The electronic device uses the running data query engine to obtain a query result based at least on the at least one query field, the identifier of the target data model and the at least one output field.

[0077] The embodiment of the present application does not specifically limit the form of the data query engine, and can be configured according to actual needs. For example, the data query engine can be a Presto high-speed engine, a Spark analysis engine, or Hive, etc.

[0078] S303 can be implemented as follows: the electronic device utilizes the running data query engine to first search for the target data model pointed to by the identifier of the target data model, then search for the values ​​of each field corresponding to at least one query field in the target data model, then determine the output field according to the query field, and determine the value of the output field according to the value of the query field, and finally use at least one output field and the value of the output field as the query result.

[0079] Based on Example A, the query result obtained by the electronic device after S303 may be the following Table 3.

[0080] Table 3. Query result example

[0081] Student Name Language scores Zhang San 85 Li Si 95 …… …… Wang Wu 90

[0082] The data processing scheme provided by the embodiment of the present application includes obtaining query content; determining at least one query field, an identifier of a target data model, and at least one output field based on the query content and a metadata database; the metadata database stores fields in at least two data models; and using a running data query engine, obtaining query results based at least on the at least one query field, the identifier of the target data model, and the at least one output field. In this scheme, the fields included in multiple data models are integrated into the metadata database, so that by querying the metadata database, it is possible to find answers to questions in multiple data models, with high query accuracy; and the limitation on business scenarios can be removed.

[0083] The following is a detailed description of the metadata library.

[0084] In one example, the metadata database also stores the alias of the field, the type of the field, the dimension enumeration value of the field, the field measurement value, the enumeration value mapping field, etc.; the dimension enumeration value of the field is the enumeration value of the possible values ​​of the field; the measurement value of the field is the left and right boundary values ​​of the field value; the enumeration mapping field is used to represent that an enumeration value uniquely corresponds to a field.

[0085] Exemplarily, the metadata database may also store the related contents in Table 4 below.

[0086] Table 4 Metadatabase example

[0087]

[0088]

[0089] It is understandable that the metadata library can also store FAQ sub-libraries or other data, etc., which will not be described in detail here.

[0090] The embodiment of the present application does not specifically limit the storage format of data in the metadata database, and it can be adjusted according to actual needs.

[0091] Example 1: It can be stored in the form of a file.

[0092] For example, the fields and field aliases included in the data model can be stored in one file; the identifier of the data model to which the field belongs can be stored in another file; the type of the field can be stored in another file; the attributes of the field can be stored in another file, and so on.

[0093] Example 2: It can be stored in the form of a table.

[0094] For example, various information of the fields can be organized into a table for storage.

[0095] For ease of description, at least one query field, an identifier of a target data model, and at least one output field are referred to as a first query condition below.

[0096] In one example, the first query condition may be equivalent to the structured query condition corresponding to the first query field.

[0097] In another example, the structured query condition corresponding to the first query field may also include other contents. For example, the structured query condition corresponding to the first query field may also include an attribute screening condition, an aggregation function of a metric field, and the like.

[0098] S302 When the electronic device determines the first query condition, it may include but is not limited to the following two implementation methods:

[0099] Implementation A: determining the first query condition according to the FAQ sub-database in the metadata database;

[0100] Implementation method B: determining the first query condition according to the field in the metadata database and the identifier of the data model to which the field belongs.

[0101] Implementation A may include: a plurality of question contents and query conditions corresponding to the question contents are pre-stored in the FAQ sub-library in the metadata library. Implementation A may be implemented as follows: the electronic device matches the query content with the question contents in the FAQ sub-library, and if there is a target question content satisfying the quick query condition in the FAQ sub-library, the query condition corresponding to the target question content is read, and the query condition corresponding to the target question content is used as the first query condition.

[0102] The embodiment of the present application does not specifically limit the content of the quick query condition, and can be configured according to actual needs. For example, the quick query condition may include: the content similarity between the query content and the query content is greater than or equal to the quick query threshold.

[0103] When the first query condition is determined by implementation method A, since the FAQ sub-library pre-stores multiple question contents and query conditions corresponding to the question contents, when determining the first query condition, there is no need to perform a detailed analysis on the first query condition to generate the first query condition, and the query condition corresponding to the question content that meets the quick query condition in the FAQ sub-library can be directly called as the first query condition. In this way, the response speed is improved and the user experience is further enhanced.

[0104] It is understandable that the FAQ sub-database may also be stored as an independent database in parallel with the metadata database, that is, the first query condition may also be determined by accessing the independent FAQ database.

[0105] It can be understood that if there is no target question content satisfying the quick query condition in the FAQ sub-database, implementation method B can be used to determine the first query condition.

[0106] Implementation B may include but is not limited to the following Figure 4 S401 to S404 shown.

[0107] S401: The electronic device determines the at least one query field based on the query content and the candidate field set in the metadata database.

[0108] The candidate field set includes fields in at least two data models, and alias fields of the fields.

[0109] The embodiment of the present application does not specifically limit the number of query fields, which can be determined according to actual needs.

[0110] S401 may be implemented as follows: based on the query content, the electronic device determines the fields in the candidate set in the metadata that meet the field screening condition as the query field.

[0111] The embodiment of the present application does not specifically limit the field screening conditions and can be configured according to actual needs.

[0112] S402: The electronic device searches the metadata database for an identifier of a data model to which each query field in the at least one query field belongs, as an identifier of at least one query data model.

[0113] Since the metadata database also stores the identifier of the data model to which the field belongs, S402 can be implemented as follows: the electronic device searches the metadata database for the identifier of the data model to which each query field belongs, as the identifier of the query data model.

[0114] Based on Example A, S402 can be implemented as Example B: the electronic device finds in the metadata database that the data model belonging to the field student name includes data model A and data model B, and the data model belonging to the field Chinese score includes data model A. Therefore, the query data model determined by the electronic device includes: data model A and data model B.

[0115] S403: The electronic device determines an identifier of the target data model according to an identifier of the at least one query data model.

[0116] S403 can be implemented as follows: the electronic device determines the relationship between the number of identifiers of at least one query data model and a first numerical value; if the number of identifiers of at least one query data model is equal to the first numerical value, the identifier of the query data model is used as the identifier of the target data model; if the number of identifiers of at least one query data model is greater than the first numerical value, the identifier of the query data model that meets the constraint condition is used as the identifier of the target data model.

[0117] The embodiment of the present application does not limit the specific content of the constraint condition, and can be configured according to actual needs. For example, the constraint condition may include: the identifier of the query data model with the largest number of fields.

[0118] The embodiment of the present application does not impose a unique limitation on the specific value of the first value, and can be configured according to actual needs. In one example, the first value can be 1.

[0119] Based on example B, assuming that the first value is 1, the constraint conditions include the identification of the data model including: the identification of the data model with the largest number of included fields, S403 can be implemented as example C: the electronic device determines that the number of identifications of the queried data model is 2, and since 2 is greater than the first value 1, it further determines that the number of fields included in data model A is 2, and the number of fields included in data model B is 1, thereby determining that the number of fields included in data model A is greater than the number of fields included in data model B, and therefore determines that the identification of data model A is the identification of the target data model.

[0120] It should be noted that if the number of the identifiers of at least one query data model is less than the first value, the identifier of at least one query data model is used as the identifier of the target data model.

[0121] S404: The electronic device determines the at least one output field based on the at least one query field.

[0122] The embodiment of the present application does not specifically limit the number of output fields, which can be determined based on time requirements.

[0123] In an example, S404 may be implemented as: the electronic device uses all fields in the at least one query field as output fields.

[0124] In another example, S404 may be implemented as: the electronic device deletes a field that meets the optimization condition in at least one query field, thereby using the remaining query fields after the deletion as output fields.

[0125] The optimization conditions of the embodiments of the present application are not specifically limited and can be configured according to actual needs.

[0126] In one example, the implementation of S401 may include but is not limited to the following S4011 to S4014.

[0127] S4011. The electronic device obtains at least one content slice included in the query content.

[0128] The electronic device removes stop words from the query content, extracts valuable text information from the query content, and then uses a content slicing algorithm to slice the query content to obtain one or more content slices.

[0129] Based on Example A, S4011 can be implemented as Example D: the electronic device removes stop words in the query content to obtain "student Chinese scores", and then uses a content slicing algorithm to obtain two content slices: "student" and "Chinese scores".

[0130] S4012. The electronic device calculates, for each content slice in the at least one content slice, the content similarity between the content slice and each candidate field in the candidate field set.

[0131] S4012 can be implemented as follows: the electronic device performs the following operations on each content slice in at least one content slice. Taking the first content slice (any content slice) as an example, a similarity algorithm is used to calculate the content similarity between the first content slice and each candidate field in the candidate set. The electronic device traverses each content slice in at least one content slice to obtain the content similarity between each content slice and all candidate fields.

[0132] Based on Example D, S4012 can be implemented as Example E: the electronic device adopts a similarity algorithm to calculate the content slice "student", and the content similarities between it and the candidate fields "student name", "Chinese score", "mathematics score", "English score", "physical education score", "gender", "height", "home address" and "health status" are 0.6, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, and 0.1 respectively; the content slice "Chinese score" is calculated, and the content similarities between it and the candidate fields "student name", "Chinese score", "mathematics score", "English score", "physical education score", "gender", "height", "home address" and "health status" are 0.1, 1, 0.4, 0.4, 0.4, 0.1, 0.1, 0.1, and 0.1 respectively.

[0133] S4013. The electronic device determines a target similarity among the content similarities that satisfies a first similarity condition.

[0134] The embodiment of the present application does not limit the specific content of the first similarity condition, and can be configured according to actual needs. For example, the first similarity condition may include that the content similarity is greater than or equal to a content similarity threshold.

[0135] S4013 may be implemented as follows: the electronic device traverses all content similarities, and determines the content similarities that meet the first similarity condition as the target similarity.

[0136] Based on Example E, the first similarity condition is that the content similarity is greater than or equal to 0.5; S4013 can be implemented as Example F: the electronic device determines that the content similarity 0.6 between the content slice "student" and the candidate field "student name" is greater than the content similarity threshold 0.5; determines that the content similarity 1 between the content slice "Chinese score" and the candidate field "Chinese score" is greater than the content similarity threshold 0.5, so the electronic device determines the target similarity including: the content similarity between the content slice "student" and the candidate field "student name", and the content similarity between the content slice "Chinese score" and the candidate field "Chinese score".

[0137] S4014: The electronic device determines the candidate field that obtains the target similarity as the query field.

[0138] The electronic device obtains the candidate field for determining the target similarity as the query field.

[0139] Based on example F, after S4014, the electronic device determines that the query fields include "student name" and "Chinese language score".

[0140] After the electronic device determines at least one query field in S401, the query field may be removed. The removal process is described below by taking the removal of the first field as an example.

[0141] The electronic device determines whether the second field exists in the query field, and if: the first field covers the second field, and the content similarity of the first field is less than the content similarity of the second field, then removes the first field from the query field.

[0142] The first field is different from the second field; the content similarity of the first field is the highest content similarity between the first field and at least one content slice; the content similarity of the second field is the highest content similarity between the second field and at least one content slice.

[0143] For example, if at least one query field includes a date field with a similarity of 0.88 and a birthday field with a similarity of 0.91, the date field should be removed.

[0144] After the electronic device determines at least one query field in S401, it can also add other query fields. The process of adding query fields is described below, which may include but is not limited to: Figure 5 S501 to S505 shown.

[0145] S501: The electronic device determines unrecognized text in the query content.

[0146] The unrecognized text is composed of the second slice in the query content except the first slice, and the first slice is the slice that obtains the target similarity. In other words, the first slice is the recognized slice, and the second slice is the unrecognized slice.

[0147] S501 can be implemented as follows: the electronic device obtains the query content, determines a content slice related to the target similarity as the first slice, and deletes the first slice in the query content, thereby obtaining unrecognized text in the query content.

[0148] S502: The electronic device converts the unrecognized text into pinyin.

[0149] S502 may be implemented as follows: the electronic device converts the unrecognized text into pinyin using a text recognition algorithm and a pinyin conversion algorithm.

[0150] S503: The electronic device obtains at least one pinyin slice included in the pinyin.

[0151] The implementation process of S503 may be: the electronic device processes the pinyin converted from the unrecognized text using a pinyin slicing algorithm, thereby obtaining one or more pinyin slices.

[0152] S504. The electronic device calculates the pinyin similarity between each pinyin slice in the at least one pinyin slice and each candidate field in the candidate field set.

[0153] The specific implementation of S504 can refer to the detailed description of the electronic device calculating the content similarity between the content slice and each candidate field in the candidate field set in S4012, which is not repeated here.

[0154] S505: The electronic device adds the candidate fields whose pinyin similarity satisfies the second similarity condition to the at least one query field.

[0155] The embodiment of the present application does not limit the specific content of the second similarity condition, and can be configured according to actual needs. For example, the second similarity condition may include that the content similarity is greater than or equal to the pinyin similarity threshold.

[0156] The embodiment of the present application does not specifically limit the magnitude relationship between the first similarity threshold and the second similarity threshold. In one example, the first similarity threshold may be less than or equal to the second similarity threshold.

[0157] The specific implementation of S505 may refer to the detailed description of the electronic device determining the query field in S4013 to S4014, which will not be described in detail here.

[0158] In this way, the query field can be expanded through relevant calculations of pinyin similarity, which improves the accuracy and comprehensiveness of the query field, thereby further improving the accuracy of the query.

[0159] If the query field includes a metric field with a metric attribute, the data processing method provided in the embodiment of the present application also needs to obtain the aggregation function corresponding to the metric field. This process may include but is not limited to the following Figure 6 S601 to S602 shown.

[0160] S601. The electronic device obtains a metric field having a metric attribute in the at least one query field.

[0161] The metric attribute is used to characterize the size of the field value.

[0162] S601 may be implemented as follows: the electronic device searches for the attribute of each query field in the metadata database, thereby determining a metric field having a metric attribute.

[0163] Based on Example E, the query fields include "student name" and "Chinese language score". The electronic device determines in the metadata database that the attribute of the field "student name" is a spatial dimension attribute, which belongs to the spatial dimension field; and determines that the attribute of the field "Chinese language score" is a measurement attribute, which belongs to the measurement field.

[0164] S602: The electronic device sets an aggregation function of the metric field according to an aggregation condition.

[0165] The implementation of this application does not limit the specific content of the aggregation conditions, and can be configured according to actual needs.

[0166] In an example, the aggregation condition may include: the aggregation function of the field to which the metric can be added is a sum function (sum), and the aggregation function of the field to which the metric cannot be added is an average function (avg).

[0167] Optionally, an aggregation function may be set for a dimension field. For example, the aggregation function of the dimension field may be set to a count function (count).

[0168] Correspondingly, S303 can be implemented as follows: the electronic device utilizes the running data query engine to first search for the target data model pointed to by the identifier of the target data model, then search for the values ​​of each field corresponding to at least one query field in the target data model, then determine the output field according to the query field, and determine the value of the output field according to the value of the query field and the measurement function, and finally use the output field and the value of the output field as the query result.

[0169] The data processing method provided in the embodiment of the present application may also include determining an attribute screening condition corresponding to a field attribute. The process may include but is not limited to the following: Figure 7 S701 and S702 shown.

[0170] S701. The electronic device determines a field attribute of each field included in the at least one query field.

[0171] The electronic device searches the metadata database for field attributes of each query field.

[0172] The attributes of a field are used to characterize the characteristics of the field content (also called the field value). Different characteristics correspond to different field attributes.

[0173] For example, the content of the Chinese score field in Table 1 is a numerical value, which can be measured, so the Chinese score field has a measurement attribute;

[0174] For another example, the date of birth field in Table 1 has a time meaning, and the content of this field cannot be measured, so the date of birth field has a dimension attribute;

[0175] For another example, the student name field in Table 1 does not have a time meaning, and the content of the bullet is not measurable, so the student name field has a spatial dimension attribute.

[0176] S702: The electronic device determines an attribute screening condition corresponding to the field attribute based on the field attribute of each field.

[0177] The embodiment of the present application does not limit the specific content of the attribute screening condition corresponding to the field attribute, and can be configured according to actual needs.

[0178] In one example, the fields of the time dimension attribute correspond to the time dimension filtering condition; the fields of the space dimension attribute correspond to the space dimension filtering condition; and the fields of the measurement attribute correspond to the measurement filtering condition.

[0179] Correspondingly, S303 can be implemented as follows: the electronic device utilizes the running data query engine to first search for the target data model pointed to by the identifier of the target data model, and then searches for the values ​​of each field corresponding to the query field in the target data model according to the attribute screening conditions corresponding to each field, and then determines the output field according to the query field, and determines the value of the output field according to the value of the query field, and finally uses the output field and the value of the output field as the query result.

[0180] It can be understood that when the processing method provided in the embodiment of the present application also includes the process of obtaining the aggregation function corresponding to the metric field and determining the attribute filtering condition corresponding to the field attribute, S303 can be implemented as follows: the electronic device uses the running data query engine to first search for the target data model pointed to by the identifier of the target data model, and then searches for the values ​​of each field corresponding to the query field in the target data model according to the attribute filtering condition corresponding to each field, and then determines the output field according to the query field, and determines the value of the output field according to the value of the query field and the metric function, and finally uses the output field and the value of the output field as the query result.

[0181] The implementation of S702 may include but is not limited to obtaining the filtering conditions of the time dimension attribute, obtaining the filtering conditions of the space dimension attribute, and obtaining the filtering conditions of the measurement attribute; wherein, the process of obtaining the filtering conditions of each attribute is similar, and now taking the filtering conditions of the time dimension attribute as an example, the attribute filtering conditions corresponding to the field attribute are described, and the process may include but is not limited to the following Figure 8 S801 and S802 shown.

[0182] S801. An electronic device obtains at least one time content slice.

[0183] The time content slices correspond to fields with time dimension attributes.

[0184] S801 may be implemented as follows: the electronic device identifies a time-related content slice in the content slices as at least one time content slice.

[0185] The embodiment of the present application does not impose a unique limitation on the number of time content slices, which can be determined based on actual conditions.

[0186] Example 1: The time content slice acquired by the electronic device may include “April to October 2015”.

[0187] Optionally, the time content slice may be converted into synonyms of time and numbers, and the time phrases and colloquial numerical phrases in the time content slice may be converted into a form that is easy to query.

[0188] For example: "this year" used to express time is changed to "2021", and "last month" is changed to "June 2021"; "two" or "two" used to express numbers is changed to "2", and "thousand" is changed to "000".

[0189] S802: The electronic device converts the content of each of the at least one time content slice into a corresponding enumeration filtering function and a filtering value as an attribute filtering condition corresponding to the field attribute.

[0190] The embodiment of the present application does not specifically limit the function type of the enumeration filtering function and can be configured according to actual needs.

[0191] In an example, the enumeration screening functions may include: an interval function (area), an equal function (EQ), a not equal function (NEQ), a greater than function (GT), and a greater than or equal to function (GTEQ).

[0192] Based on Example 1, S802 may be Example 2: the electronic device determines that the enumeration filter function is area, and the filter value is [2015-04, 2015-10]; the corresponding filter condition is between'2015-04'and'2015-10'.

[0193] The data processing method provided in the embodiment of the present application can also modify the determined first query condition. The modification process may include: receiving the input modification content, and obtaining the query result according to the modification content; wherein the modification content may include at least one of the following: the identifier of the target data model, the output field. It should be noted that at this time, S302 is no longer executed, and S303 is directly executed according to the modified content to obtain the query result.

[0194] The data processing method provided in the embodiment of the present application can also organize the query results into visual graphics or data reports, and then send them to the client.

[0195] Below, the data processing method provided in the embodiment of the present application is explained through specific application scenarios.

[0196] In recent years, with the development and popularization of AI and NLP technologies, intelligent question-answering products have been increasingly widely used in e-commerce, retail, finance, transportation and other industries and in daily life. Representative products include Apple's voice assistant, Microsoft XiaoIce, and JD's intelligent customer service. These products are usually based on artificially constructed knowledge bases or knowledge graphs, and use deep learning, machine learning and other models, and semantic analysis to understand and reason about the natural language input by users, and give intuitive answers.

[0197] At the same time, with the digitalization and intelligent development trend of the BI industry and the unprecedented demand for big data mining and exploration, intelligent question answering for structured databases has attracted the attention of more and more companies and researchers, and related research and products from natural language to structured query language have sprung up. Some products have been initially put into use.

[0198] 1. From a functional perspective, interactive intelligent question-and-answer products (such as Apple's voice assistant) have the following shortcomings:

[0199] (1) Knowledge bases or knowledge graphs that are highly dependent on human construction and editing. Since the knowledge base itself occupies a large amount of storage resources, this technology will affect the read and write response efficiency. At the same time, the initial construction and subsequent maintenance costs are high, so it is difficult to adapt to the rapid update of enterprise data.

[0200] (2) It can only return the existing results in the knowledge base and cannot perform any operations or other processing on the data.

[0201] (3) It is difficult to guarantee the accuracy of the results in highly specialized and complex business scenarios.

[0202] 2. From a technical perspective, the core of interactive intelligent question-answering technology is structured query language technology, which has the following main shortcomings:

[0203] (1) This technology is still in its infancy. Current products can generally only be used to query a single data source table (also known as a single data model or a data model). For queries on multiple data source tables, the accuracy is low and far from being applicable. For example, the current optimal accuracy of Yale University's Spider English dataset can only reach 71%, which is not applicable to the query of multiple data source tables in flexible and complex actual business scenarios.

[0204] (2) Compared with English, Chinese has a complex language structure and flexible expression forms. In addition, structured query language technology faces the problems of lack of evaluation mechanism and immature model, making it more difficult to improve practical results.

[0205] The data processing method provided in the embodiment of the present application is described in detail below using a multi-data source table as an example.

[0206] The embodiments of the present application can be applied to the query of multi-data source tables in complex business scenarios at the enterprise level. An end-to-end data-driven multi-table intelligent question and answer system is designed to realize the interaction from natural language input to reports or visual graphic query results, and effectively solve the limitations of current interactive intelligent question and answer and structured query language technologies, such as low accuracy of Chinese semantic parsing, difficulty in productizing multi-table queries, and high dependence on large knowledge bases.

[0207] The structure of the multi-table intelligent question answering system designed in this application is as follows Fig. 9 The whole system consists of four parts: client 901, semantic analysis module 902, background processing module 903, and data storage module 904. The semantic analysis module 902 is the core of the system.

[0208] The user can send query content to the text input unit of the client 901, and the client 901 can display the structured query conditions corresponding to the query content to the user, and can also display the query results to the user in the form of data or visual images; the semantic parsing module 902 can include a structured query language model (NL2SQL model) application programming interface (Application Programming Interface, API) 9021, NL2SQL model 9022 and platform service (Platform as a The NL2SQL model API 9021 is used to provide an API interface to receive query content. The NL2SQL model 9022 can be used to call the data source 9041 in the data storage module 904 to convert the query content into corresponding structured query conditions. The PaaS platform 9023 is used to automatically deploy metadata knowledge. The background processing module 903 may include a FAQ library 9031, a query engine 9032, a data processing unit 9033, and a visualization unit 9034. The FAQ library 9031 stores commonly used question content and structured query conditions corresponding to the question content. The query engine 9032 is used to search for structured query conditions corresponding to the query content in the FAQ library. The data processing unit 9033 can be used to organize the query results into the form of data reports. The visualization unit 9034 can be used to organize the query results into visual graphics.

[0209] It should be noted that Fig. 9 The semantic parsing module 902 in may be equivalent to Figure 2 The semantic parsing module 201 in Fig. 9 The background processing module 903 in the Figure 2 The station processing module 203, Fig. 9 The data storage module 904 in may be equivalent to Figure 2The data storage module 202 in Fig. 9 The semantic analysis module 902, the background processing module 903 and the data storage module 904 in the Figure 2 The data processing end 20 in.

[0210] The workflow of this system includes Fig.10 The steps shown include data preparation, user input, client response, text semantic parsing, background processing, result output, and user interaction.

[0211] Now combined Fig. 9 and Fig.10 , the data processing process provided by the implementation of this application is described, which may include but is not limited to the following S1 to S7.

[0212] S1: Data preparation.

[0213] The electronic device writes multiple data models required for query in various business scenarios into the data storage module in the form of a two-dimensional table (also called a data source table) to generate a data source. The data storage medium can be a distributed file system, a column database, a relational database, etc.

[0214] S2: User input.

[0215] The user enters the query content (also called input text) in the text input unit of the client, for example, "the number of people who have resigned from Department A in the past three months"; wherein the client can be in the form of a web page or an app, and the text input unit can be in the form of Fig.11 shown.

[0216] S3: Client response.

[0217] After receiving the input query content, the client first matches the question content in the FAQ library with the preset similarity threshold; if there is no question that meets the matching conditions in the FAQ library, the intelligent computing unit is required to perform text parsing, that is, execute S3; if the question with the highest similarity in the FAQ library meets the matching conditions with the input text, it responds quickly, returns the structured query conditions corresponding to the question in the FAQ library, and executes S4.

[0218] The structured query conditions can be SQL (whether the query engine supports it) statements, which are convenient for the data query engine to execute directly. In order to facilitate users to modify the query conditions for extended queries, or when there is an internal protocol between the data source and the background processing module that the intelligent algorithm module cannot obtain, the structured query conditions can be set in the form of data objects. For example, the structured query conditions can contain 4 parts of information: 1) the name or code of the data source table to be queried (which can be called the identifier of the data model); 2) query (SELECT) fields and output (GROUP BY) dimension fields; 3) metric fields, and the aggregation functions of each metric field; 4) the field name of the filter condition (i.e., WHERE condition) part and its filter function and filter value.

[0219] S4: Text semantic parsing.

[0220] The client sends a request to the API of the semantic parsing module. The core of the semantic parsing module is the NL2SQL algorithm model, which is deployed on the PaaS cloud service platform and provides a calling interface to the client.

[0221] The algorithm model consists of two parts: metadata knowledge base (also called metadata database) and text parsing algorithm. The metadata knowledge base stores offline information such as field data type, field type, field table, dimension enumeration value, metric boundary value, field alias, enumeration value mapping field, etc. in the data source table in the form of key-value pairs. The specific content is shown in Table 5.

[0222] Table 5. Metadata knowledge base content example

[0223]

[0224]

[0225] Since metadata information may be updated over time, the online model of this system regularly reads the latest metadata knowledge from the data source and automatically updates the online model using a workflow scheduling task system. This automatically updated metadata knowledge base mechanism effectively reduces storage space requirements, improves read and write response efficiency, and ensures the timeliness of query results.

[0226] The steps of semantic parsing of input text by the NL2SQL model are described in detail in S4.1 to S4.5 below.

[0227] S4.1. Remove stop words.

[0228] Remove stop words from the query content entered by the user.

[0229] S4.2. Identify the fields.

[0230] The specific steps may include the following S4.2.1 to S4.2.4.

[0231] S4.2.1. Preliminary identification of fields.

[0232] All fields and their aliases in the metadata knowledge base are taken as candidate sets, and edit distance matching or named entity recognition technology is used. With a preset similarity threshold (for example, the edit distance similarity threshold is set to 0.85) as the standard, all successfully matched fields are obtained as a list of identified fields, and their field names, the slices with the highest similarity in the text (i.e., substrings, hereinafter referred to as matching slices) and their similarity values ​​are recorded.

[0233] S4.2.2. Remove the covered fields.

[0234] In the identified field list, if a field name is completely covered by another successfully matched field name and its similarity is lower than the latter, remove the field from the identified field list. For example, in the identified field list, there is a field date with a similarity of 0.88, and if there is also a field birth_date with a similarity of 0.91, the field date should be removed.

[0235] S4.2.3. Hide matching slices.

[0236] After confirming the identified fields through the above steps, hide the corresponding matching slices of these fields in the input text to avoid interfering with the subsequent steps.

[0237] S4.2.4, supplement the matching of similar-sounding characters.

[0238] In order to enhance the tolerance to typos and misspellings, the Chinese characters in the processed text and candidate field names are converted into pinyin, and the edit distance method is used for supplementary matching (the similarity threshold can be appropriately increased compared to S4.2.1 to ensure the accuracy of the results), and the successfully matched fields are added to the list of identified fields.

[0239] S4.3. Identify field filter conditions.

[0240] Specifically, it may include but is not limited to the following 3.3.1 to S4.3.5.

[0241] S4.3.1. Synonymous conversion between time and numbers.

[0242] Convert time-sensitive time phrases and colloquial numerical phrases in the input text into structured forms that are easy to query. For example, "this year" expressing time is converted to "2021", and "last month" is converted to "March 2021"; "two" or "two" expressing numbers is converted to "2", and "thousand" is converted to "000".

[0243] S4.3.2. Identification of filtering conditions for time dimension fields.

[0244] Using regular expression matching and other methods, the time dimension fields in the identified fields are identified in the input text in sequence for the three types of statements in S4.3.2.1 to S4.3.2.3. If there are multiple time dimension fields, the left and right boundary values ​​of the time dimension fields in the "Dimension Enumeration Value" of the metadata knowledge base are combined to select the field where the filter value exists. After each step of identification is completed, the successfully matched slices in the input text are hidden.

[0245] S4.3.2.1. Identification of multiple enumeration filter functions and values.

[0246] Related candidate functions include: area (between...and), greater than or equal to function (GTEQ), greater than function (GT), less than or equal to function (LTEQ), less than function (LT), etc. For example, from the slice "2015 April to October", it can be identified that the filtering function is area, and the filtering value is ['2015-04','2018-10'], that is, the filtering condition is between'2015-04'and'2018-10'.

[0247] S4.3.2.2. Identification of single enumeration-valued functions and values.

[0248] Related functions include: equal function (EQ), not equal function (NEQ), etc. For example, from the slice "Except 2020", it can be identified that the filter function is NEQ, the filter value is '2020', and the filter condition is = '2020'.

[0249] S4.3.2.3. Identification of time-sensitive descriptive phrase screening functions and values.

[0250] Related candidate functions include: GTEQ, GT, etc. For example, the slice "recent three years" can identify the screening function as GTEQ, the screening value is '2018' (assuming the current year is 2021), and the screening condition is >= '2018'.

[0251] S4.3.3, Identify the metric field filter conditions.

[0252] Similar to the time dimension field, regular expression matching and other methods are used to identify the metric fields in the identified fields, and the two types of statements in S4.3.3.1 to S4.3.3.2 are identified in the input text in sequence. If there are multiple metric fields, the left and right boundary values ​​of the metric fields in the metadata knowledge base "metric boundary value" are combined to select the field where the filter value exists. After each step of identification is completed, the successfully matched slices in the input text are hidden.

[0253] S4.3.3.1. Identification of multiple enumeration filter functions and values.

[0254] Relevant candidate functions include: area(between…and), GTEQ, GT, LTEQ, LT, etc.

[0255] S4.3.3.2. Identification of single enumeration-valued functions and values.

[0256] Relevant candidate functions include: EQ, NEQ, etc.

[0257] S4.3.4, identification of spatial dimension field screening conditions.

[0258] Specifically, it may include but is not limited to the following S4.3.4.1 to S4.3.4.3.

[0259] S4.3.4.1. For the spatial dimension fields in the identified fields, the “dimension enumeration value” in the metadata knowledge base is combined to match its enumeration value with a preset similarity threshold.

[0260] S4.3.4.2. For the above successfully matched enumeration value list, use the method of S4.2.3 to remove the covered enumeration values ​​and obtain the final screening value.

[0261] S4.3.4.3. Use regular matching and other methods, combined with the screening value obtained in the previous step, to determine the screening function. The relevant candidate functions include: EQ, NEQ, IN, and NIN. Hide the successfully matched slices in the input text.

[0262] S4.3.5. Supplement the identification of spatial dimension fields through enumeration values.

[0263] S4.3.2 to S4.3.4 identify fields by field name or alias. In real query scenarios, users may omit field names in the filter conditions they enter and only enter enumeration values. For example, "Performance distribution of employees in Department A" implies the filter condition "Department name = 'Department A'", and both the enumeration value and the corresponding field need to be identified. Note that only the enumeration values ​​contained in the "Enumeration value mapping field" of the metadata knowledge base can be identified here. If the enumeration value appears in multiple fields, it will not be identified.

[0264] Specifically, it may include but is not limited to S4.3.5.1 to S4.3.5.3.

[0265] S4.3.5.1. Use the edit distance method or other methods to match the input text with the enumeration values ​​in the "enumeration value mapping field" at a pre-set similarity threshold, and record all successfully matched enumeration values ​​and their corresponding field names and matched input content slices.

[0266] S4.3.5.2. Remove covered enumeration values. In the list of identified enumeration values, if an enumeration value name is completely covered by another successfully matched enumeration value name and its similarity is lower than the latter, then remove the enumeration value from the list of identified enumeration values.

[0267] S4.3.5.3. Use regular matching and other methods, combined with the filter enumeration value obtained in the previous step, to determine the filter function. The relevant candidate functions include: EQ, NEQ, IN, and NIN. Add the field corresponding to the enumeration value to the list of identified fields. Hide the successfully matched slices in the input text.

[0268] S4.4. Identify the data source table.

[0269] For multi-table queries, after identifying the fields and filter conditions, you need to collect the "field table" in the metadata knowledge base to further identify the data source table to be queried. The specific steps are as follows.

[0270] S4.4.1. If the list of identified fields is empty, return an exception message "No valid field identified" to the API. Otherwise, execute step S4.4.2.

[0271] S4.4.2. If there is a unique data source table that contains all the identified fields, this table is determined to be the query data source table; otherwise, if the number of identified fields contained in a data source table exceeds the sum of the number of identified fields contained in other data source tables, this table is used as the query data source table, and the results related to the fields that are not in this table are removed from the identified fields and the filter conditions; otherwise, an exception message "Unable to determine the data source table" is returned to the API.

[0272] S4.5. Optimization of recognition results.

[0273] Perform final optimization and structural integration on the identified results and return them to the API and client. The specific steps are as follows S4.5.1-S4.5.4.

[0274] S4.5.1. Improve the screening conditions.

[0275] Some data source tables need to be supplemented with default filtering conditions. For example, if the query data source table identified in the previous step is a zipper table or a partition table, you need to add default filtering conditions to make the query range valid data or current data. In addition, some data tables may contain some dirty data, which needs to be removed by adding default filtering conditions.

[0276] S4.5.2. Optimize output fields.

[0277] In the list of fields identified in the previous step, some fields may be fields that users only want to use for filtering and do not want to see in the query results to avoid interference. This requires the system to make a final determination of the output fields, identify the fields that are only used for filtering and do not output them.

[0278] The output field consists of output dimensions and output metrics, which are initialized to the dimension field and metric field in the identified field respectively. The following is a recommended output field selection logic: for the identified time dimension field, if it is in the filter condition and the filter function is EQ, then the time dimension is removed from the output dimension (that is, the time dimension is only used for filtering, the same below); for the identified space dimension field, if it is in the filter condition and the filter function is not IN, then the space dimension is removed from the output dimension; for the identified metric field, if it is in the filter condition, then the metric field is removed from the output metric.

[0279] S4.5.3. Determine the aggregation function of the metric field.

[0280] The output data is summary data, so the aggregation function of the metric field must be returned. This system sets the aggregation function to sum for additive metrics and avg for non-additive metrics. If there is no metric field in the output field, add the output aggregation indicator: count.

[0281] S4.5.4. Integrate the above identification results, including the data source table, output dimension field list, output metric field and its aggregation function, filter conditions (fields, filter functions, filter values) and other information, return them to the API in the form of structured query conditions, and then output them to the client.

[0282] S5. Background processing.

[0283] The data query engine obtains structured query conditions from the client, executes the corresponding SQL (or Hive SQL) statements, and reads data from the data source. The query can be performed using the Presto high-speed engine, Spark analysis engine, Hive, etc., among which Presto is faster. After reading in the relevant summary data, the data processing unit performs necessary processing on it and integrates it into the form of output required reports or visual graphics.

[0284] S6. Output the results.

[0285] The background will output the processed data results to the client in the form of reports or visual graphics.

[0286] S7. User interaction.

[0287] The user can modify the information in the structured query conditions and perform extended queries, such as modifying the data source table or output field, modifying the value range of the filter field, etc. At this time, the modified structured query conditions are directly transmitted to the data query engine without going through the semantic parsing module, and S5 and S6 are executed in sequence.

[0288] The technical effects of the data processing method provided by the embodiment of the present application may include:

[0289] 1. Aiming at the database query requirements in complex enterprise-level business scenarios, an end-to-end data-driven multi-table intelligent question-answering system is designed, which consists of a client, a semantic parsing module, a background processing module, and a data storage module. Through user input, client response, text semantic parsing, background processing, result output, user interaction and other steps, it realizes the output and interaction of reports or visual graphics from natural language input to data. Among them, the semantic parsing module is the core of this system, which effectively solves the limitations of current intelligent question-answering and NL2SQL technologies, such as low accuracy of Chinese semantic parsing, difficulty in productizing multi-table queries, and high dependence on large knowledge bases; it only stores metadata knowledge bases and automatically deploys them regularly, which effectively saves storage space, improves the efficiency of reading and writing, and ensures the timeliness of output.

[0290] 2. The semantic parsing module designs an intelligent text semantic parsing algorithm, which completes text semantic parsing through field recognition, filter condition recognition, data source recognition, and recognition result optimization. The filter condition recognition includes time dimension filter condition recognition, measurement filter condition recognition, spatial dimension filter condition recognition, and enumeration value supplement recognition, etc., to maximize the recognition of user needs. In semantic parsing, synonym conversion, hidden matching slices, etc. are used to effectively improve the recognition accuracy.

[0291] Fig.12 is a structural diagram of a data processing device according to an embodiment of the present application, such as Fig.12 As shown, the data processing device 120 includes: an acquisition unit 1201, a determination unit 1202 and a processing unit 1203. Among them:

[0292] The acquisition unit 1201 is used to acquire the query content.

[0293] The determination unit 1202 is configured to determine at least one query field, an identifier of a target data model, and at least one output field based on the query content and a metadata database; the metadata database stores fields in at least two data models.

[0294] The processing unit 1203 is configured to obtain a query result based on at least the at least one query field, the identifier of the target data model and the at least one output field by using the running data query engine.

[0295] In some embodiments, the determining unit 1202 is further configured to:

[0296] Determine the at least one query field based on the query content and a candidate field set in the metadata database; the candidate field set includes fields in the at least two data models and alias fields of the fields;

[0297] For each query field in the at least one query field, searching the metadata database for an identifier of a data model to which the query field belongs as an identifier of at least one query data model; the metadata database also stores the identifier of the data model to which the field belongs;

[0298] Determining an identifier of the target data model according to an identifier of the at least one query data model;

[0299] The at least one output field is determined based on the at least one query field.

[0300] In some embodiments, the determining unit 1202 is further configured to:

[0301] Obtain at least one content slice included in the query content;

[0302] For each content slice in the at least one content slice, respectively, calculating the content similarity between the content slice and each candidate field in the candidate field set;

[0303] Determine a target similarity among the content similarities that satisfies a first similarity condition;

[0304] The candidate field that obtains the target similarity is determined as the query field.

[0305] In some embodiments, the data processing device 120 further includes a removal unit.

[0306] A removal unit, used for removing the first field from the at least one query field if the first field included in the at least one query field covers the second field included in the at least one query field and the content similarity of the first field is less than the content similarity of the second field; the first field is different from the second field; the content similarity of the first field is the highest content similarity between the first field and the at least one content slice; the content similarity of the second field is the highest content similarity between the second field and the at least one content slice.

[0307] In some embodiments, the data processing device 120 further includes an adding unit.

[0308] An adding unit, configured to determine unrecognized text in the query content; the unrecognized text is composed of a second slice in the query content except the first slice, the first slice being the slice for obtaining the target similarity;

[0309] Converting the unrecognized text into pinyin;

[0310] Acquire at least one pinyin slice included in the pinyin;

[0311] For each pinyin slice in the at least one pinyin slice, the pinyin similarity between the pinyin slice and each candidate field in the candidate field set is calculated respectively, and the candidate fields whose pinyin similarity satisfies the second similarity condition are added to the at least one query field.

[0312] In some embodiments, the determining unit 1202 is further configured to:

[0313] If the number of the identifiers of the at least one query data model is equal to the first value, using the identifier of the query data model as the identifier of the target data model;

[0314] If the number of the identifiers of the at least one query data model is greater than the first value, the identifiers of the query data models that meet the constraint condition are used as the identifiers of the target data model.

[0315] In some embodiments, the determining unit 1202 is further configured to: delete the field that meets the optimization condition in the at least one query field to obtain the at least one output field.

[0316] In some embodiments, the data processing device 120 further includes an aggregation unit.

[0317] an aggregation unit, used to obtain a metric field having a metric attribute in the at least one query field; the metric attribute is used to characterize that a value of the field has a size feature;

[0318] Set the aggregation function of the measurement field according to the aggregation condition;

[0319] Correspondingly, the processing unit 1203 is further used to: determine the query result based on at least the at least one query field, the identifier of the target data model, the at least one output field and the aggregation function.

[0320] In some embodiments, the data processing device 120 further includes a screening unit.

[0321] A screening unit, configured to determine a field attribute of each field included in the at least one query field;

[0322] Determine, based on the field attributes of each field, an attribute screening condition corresponding to the field attributes;

[0323] Correspondingly, the processing unit 1203 is further used to obtain the query result based on at least the at least one query field, the identifier of the target data model, the at least one output field and the attribute screening condition.

[0324] In some embodiments, the screening unit is further configured to:

[0325] Acquire at least one time content slice; the time content slice corresponds to a field having a time dimension attribute; the field attribute includes a time dimension attribute, and the time dimension attribute of the field is used to characterize that the value of the field does not have a size feature and represents a time meaning;

[0326] The content of each of the at least one time content slice is converted into a corresponding enumeration filter function and a filter value as an attribute filter condition corresponding to the field attribute.

[0327] In one example, the function of the acquisition unit 1201 can be Figure 2 The semantic parsing module 201 in the embodiment of the present invention is implemented, and the function of the determining unit 1202 can be realized by Figure 2 The semantic parsing module 201 in the embodiment of the present invention is implemented, and the function of the processing unit 1203 can be realized by Figure 2 The background processing model 203 in the embodiment of the present invention can be realized by removing units, adding units, aggregating units and filtering units. Figure 2 The semantic parsing module 201 in is implemented.

[0328] It should be noted that the data processing device provided in the embodiment of the present application includes the various units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Micro Processor Unit), a digital signal processor (DSP, Digital Signal Processor) or a field programmable gate array (FPGA, Field-Programmable Gate Array), etc.

[0329] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0330] It should be noted that in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0331] Correspondingly, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the data processing method provided in the above embodiment are implemented.

[0332] Correspondingly, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the above embodiment are implemented.

[0333] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0334] It should be noted that Fig.13 FIG. 1 is a schematic diagram of a hardware entity of the electronic device 130 according to an embodiment of the present application. In one example, the electronic device 130 may be the above-mentioned electronic device. Fig.13 As shown, the electronic device 130 includes: a processor 1301, at least one communication bus 1302, a user interface 1303, at least one external communication interface 1304 and a memory 1305. The communication bus 1302 is configured to achieve connection and communication between these components. The user interface 1303 may include a display screen, and the external communication interface 1304 may include a standard wired interface and a wireless interface.

[0335] The memory 1305 is configured to store instructions and applications executable by the processor 1301, and can also cache data to be processed or processed by the processor 1301 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0336] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0337] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0338] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0339] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0340] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0341] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0342] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0343] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method comprises: Get the query content; Determining at least one query field, an identifier of a target data model and at least one output field based on the query content and a metadata database storing fields in at least two data models; Obtaining a query result based on at least the at least one query field, the identifier of the target data model, and the at least one output field using a running data query engine; After determining at least one query field, the method further includes: Determine unrecognized text in the query content; the unrecognized text is composed of a second slice of the query content except the first slice, the first slice being the slice for obtaining the target similarity; Converting the unrecognized text into pinyin; Using a pinyin slicing algorithm to process the pinyin converted from the unrecognized text to obtain at least one pinyin slice; For each pinyin slice in the at least one pinyin slice, calculate the pinyin similarity between the pinyin slice and each candidate field in the candidate field set in the metadata database, and add the candidate fields whose pinyin similarity meets the second similarity condition to the at least one query field; wherein the candidate field set includes fields in the at least two data models, and alias fields of the fields.

2. The method according to claim 1, characterized in that The determining, based on the query content and the metadata database, at least one query field, an identifier of a target data model, and at least one output field comprises: Determining the at least one query field based on the query content and the set of candidate fields in the metadata database; For each query field in the at least one query field, searching the metadata database for an identifier of a data model to which the query field belongs as an identifier of at least one query data model; the metadata database also stores the identifier of the data model to which the field belongs; Determining an identifier of the target data model according to an identifier of the at least one query data model; The at least one output field is determined based on the at least one query field.

3. The method according to claim 2, characterized in that The determining the at least one query field based on the query content and the candidate field set in the metadata database includes: Obtain at least one content slice included in the query content; For each content slice in the at least one content slice, respectively, calculating the content similarity between the content slice and each candidate field in the candidate field set; Determine the target similarity among the content similarities that satisfies a first similarity condition; The candidate field that obtains the target similarity is determined as the query field.

4. The method according to claim 3, characterized in that The method further comprises: If the first field included in the at least one query field covers the second field included in the at least one query field, and the content similarity of the first field is less than the content similarity of the second field, the first field is removed from the at least one query field; the first field is different from the second field; the content similarity of the first field is the highest content similarity between the first field and the at least one content slice; the content similarity of the second field is the highest content similarity between the second field and the at least one content slice.

5. The method according to claim 2, characterized in that: The determining the identifier of the target data model according to the identifier of the at least one query data model comprises: If the number of the identifiers of the at least one query data model is equal to the first value, using the identifier of the query data model as the identifier of the target data model; If the number of the identifiers of the at least one query data model is greater than the first value, the identifiers of the query data models that meet the constraint condition are used as the identifiers of the target data model.

6. The method according to claim 2, characterized in that The determining the at least one output field based on the at least one query field comprises: The fields satisfying the optimization condition in the at least one query field are deleted to obtain the at least one output field.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a metric field having a metric attribute in the at least one query field; the metric attribute is used to characterize that a value of the field has a size feature; Set the aggregation function of the measurement field according to the aggregation condition; The obtaining of the query result based at least on the at least one query field, the identifier of the target data model and the at least one output field comprises: The query result is determined based at least on the at least one query field, the identification of the target data model, the at least one output field, and the aggregate function.

8. The method according to claim 1, characterized in that The method further comprises: Determining field properties of each field included in the at least one query field; Determine, based on the field attributes of each field, an attribute screening condition corresponding to the field attributes; The obtaining of the query result based at least on the at least one query field, the identifier of the target data model and the at least one output field comprises: The query result is obtained based at least on the at least one query field, the identifier of the target data model, the at least one output field, and the attribute screening condition.

9. The method according to claim 6, characterized in that The field attributes include a time dimension attribute, and the time dimension attribute of the field is used to characterize that the value of the field does not have a size feature and represents a time meaning. The determining of the attribute screening condition corresponding to the field attribute based on the field attribute of each field includes: Obtain at least one time content slice; the time content slice corresponds to a field having a time dimension attribute; The content of each of the at least one time content slice is converted into a corresponding enumeration filter function and a filter value as an attribute filter condition corresponding to the field attribute.

10. A data processing device, characterized in that: The device comprises: An acquisition unit, used for acquiring query content; a determination unit, configured to determine at least one query field, an identifier of a target data model, and at least one output field based on the query content and a metadata database storing fields in at least two data models; A processing unit, configured to obtain a query result based on at least the at least one query field, the identifier of the target data model and the at least one output field by using a running data query engine; The determination unit is further used to determine unrecognized text in the query content; the unrecognized text is composed of a second slice in the query content except the first slice, and the first slice is a slice for obtaining a target similarity; converting the unrecognized text into pinyin; using a pinyin slicing algorithm to process the pinyin converted from the unrecognized text to obtain at least one pinyin slice; for each pinyin slice in the at least one pinyin slice, calculating the pinyin similarity between the pinyin slice and each candidate field in the candidate field set in the metadata database, and adding the candidate fields whose pinyin similarity meets the second similarity condition to the at least one query field; wherein the candidate field set includes fields in the at least two data models, and alias fields of the fields.

11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the data processing method according to any one of claims 1 to 9 when executing the program.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 9 is implemented.

Images