Data processing method and device based on multi-table association

Abstract

The present application provides a data processing method and device based on multi-table association, which relates to the field of data processing technology and solves the technical problem of low accuracy of data query results. The method includes: performing semantic recognition based on the query content, and determining the range of data tables to be queried based on the association relationship between multiple data tables according to the semantic recognition results; for each target data table in the data table range, determining the target level, target type, target subject, and the number of core business fields in the target data table; determining the weight of the target data table according to the number of levels of the target level, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields; based on multiple weights corresponding to multiple target data tables, selecting the first target data table with the largest weight value, and determining the first target data table as the main query table.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and device based on multi-table association. Background Art

[0002] At present, in terms of data warehouse applications, technologies and products need to be integrated and connected, that is, products perform specific data queries based on query requests entered by users. In the process of performing data queries based on query requests entered by users, the product can clearly identify the data content, but cannot find the queried data. In this case, the data table to be queried can be obtained, but the product cannot clearly identify which data table to use among multiple data tables. The existing method is to perform data queries in data tables based on the table connection relationships preset in advance between multiple data tables, but the data query results of this method are less accurate. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing method and device based on multi-table association to solve the technical problem of low accuracy of data query results.

[0004] In a first aspect, an embodiment of the present application provides a data processing method based on multi-table association, the method comprising:

[0005] Get the query content entered by the user;

[0006] Performing semantic recognition based on the query content to obtain a semantic recognition result, and determining a range of data tables to be queried using an association relationship between multiple data tables according to the semantic recognition result;

[0007] For each target data table in the data table range, determine the target level, target type, target subject and the number of core business fields in the target data table; wherein the level includes the number of levels from low to high; the type includes the fact table class, the dimension table class, the temporary table class and the intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, and the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class; the subject includes business subjects and non-business subjects, and the subject value of the business subject is greater than the subject value of the non-business subject;

[0008] The weight of the target data table is determined according to the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields; wherein the greater the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields, the greater the weight of the target data table;

[0009] Based on the multiple weights corresponding to the multiple target data tables, selecting a first target data table having the largest value of the weight from the multiple target data tables, and determining the first target data table as a main query table;

[0010] According to the semantic recognition result, data query is performed through the main query table to obtain data query results corresponding to the query content.

[0011] In a possible implementation, the type coefficient corresponding to the intermediate table class is zero; the step of determining the weight of the target data table according to the number of levels of the target level, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields includes:

[0012] The weight of the target data table is determined by a specified weight calculation formula according to the number of levels of the target level, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields; the specified weight calculation formula is:

[0013] (a+c+d)*b;

[0014] Among them, a is the level number of the target level; c is the topic value of the target topic; d is the number of core business fields; and b is the type coefficient corresponding to the target type.

[0015] In a possible implementation, the step of selecting a first target data table having the largest value of the weight from the multiple target data tables based on the multiple weights corresponding to the multiple target data tables, and determining the first target data table as the main query table includes:

[0016] Based on the multiple weights corresponding to the multiple target data tables, selecting a first target data table having the largest value of the weight from the multiple target data tables; the first target data table includes a first target field name;

[0017] If the values ​​of the multiple weights corresponding to the multiple first target data tables are the same, the first target field name corresponding to each of the multiple first target data tables is respectively input into the natural language vectorization model to obtain the data table vector value corresponding to each of the first target data tables;

[0018] Inputting the semantic recognition result into a natural language vectorization model to obtain a query vector value corresponding to the query content;

[0019] Based on the multiple data table vector values ​​corresponding to the multiple first target data tables, a first target data table among the multiple first target data tables whose data table vector value is closest to the query vector value is determined as a main query table.

[0020] In a possible implementation, each of the data tables corresponds to a table service name, and the data table includes a field name and table content; the association relationship between the multiple data tables includes: a first mapping relationship between the table service name and the table content, and a second mapping relationship between the field name and the table content;

[0021] The step of determining the range of the data tables to be queried by utilizing the association relationship between the multiple data tables according to the semantic recognition result comprises:

[0022] Determining target table content related to the semantic content of the semantic recognition result according to the semantic recognition result;

[0023] Determine the target table service name corresponding to the target table content by using the first mapping relationship, and determine the target field name corresponding to the target table content by using the second mapping relationship;

[0024] Based on the target table business name and the target field name, the target table primary key is inferred and verified through data sampling, and the union of multiple target table primary keys is associated with multiple target data tables;

[0025] The data table range to be queried is determined according to the multiple target data tables.

[0026] In a possible implementation, before the step of determining the target field name corresponding to the target table content by using the second mapping relationship, the method further includes:

[0027] Inputting the semantic recognition result into a specified neural network model, so as to extract and output relevant data tables, query columns and screening conditions associated with the semantic recognition result through the specified neural network model;

[0028] The second mapping relationship is adjusted based on the related data table, the query column and the screening condition to optimize the association relationship between the multiple data tables through the specified neural network model.

[0029] In a possible implementation, the step of performing a data query through the main query table according to the semantic recognition result to obtain a data query result corresponding to the query content includes:

[0030] According to the semantic recognition result, data query is performed through the main query table, the query column and the screening condition to obtain an SQL statement query result;

[0031] Performing SQL statement verification on the SQL statement query result to obtain an SQL statement verification result, and obtaining a natural language query result corresponding to the query content based on the SQL statement verification result.

[0032] In a possible implementation, before the step of determining the target table service name corresponding to the target table content by using the first mapping relationship, the method further includes:

[0033] The target table business name is determined according to a naming rule combining a data warehouse keyword, a business primary key, and a time keyword.

[0034] In a second aspect, a data processing device based on multi-table association is provided, comprising:

[0035] The acquisition module is used to obtain the query content input by the user;

[0036] An identification module, used for performing semantic identification based on the query content to obtain a semantic identification result, and determining a range of data tables to be queried by using an association relationship between multiple data tables according to the semantic identification result;

[0037] The first determination module is used to determine, for each target data table in the data table range, the target level, the target type, the target subject and the number of core business fields in the target data table; wherein the level includes the number of levels from low to high; the type includes the fact table class, the dimension table class, the temporary table class and the intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, and the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class; the subject includes business subjects and non-business subjects, and the subject value of the business subject is greater than the subject value of the non-business subject;

[0038] A second determination module is used to determine the weight of the target data table according to the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields; wherein, the greater the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields, the greater the weight of the target data table;

[0039] A selection module, configured to select a first target data table having the largest value of the weight from the multiple target data tables based on the multiple weights corresponding to the multiple target data tables, and determine the first target data table as a main query table;

[0040] A query module is used to perform data query through the main query table according to the semantic recognition result to obtain data query results corresponding to the query content.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method described in the first aspect above.

[0043] The embodiments of the present application bring the following beneficial effects:

[0044] A data processing method and device based on multi-table association provided by an embodiment of the present application can obtain query content input by a user; perform semantic recognition based on the query content to obtain a semantic recognition result, and determine the range of data tables to be queried based on the association relationship between multiple data tables according to the semantic recognition result; for each target data table in the data table range, determine the target level of the target data table, the target type to which it belongs, the target subject to which it belongs, and the number of core business fields in the target data table; wherein the level includes the number of levels from low to high; the type includes a fact table class, a dimension table class, a temporary table class, and an intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class The type coefficient corresponding to the class; the subject includes a business subject and a non-business subject, and the subject value of the business subject is greater than the subject value of the non-business subject; the weight of the target data table is determined according to the number of levels of the target hierarchy, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields; wherein, the larger the number of levels of the target hierarchy, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields, the larger the weight of the target data table; based on the multiple weights corresponding to the multiple target data tables, the first target data table with the largest weight value is selected from the multiple target data tables, and the first target data table is determined as the main query table; according to the semantic recognition result, data query is performed through the main query table to obtain the data query result corresponding to the query content. In this solution, semantic recognition is used to first roughly determine the range of the data table to be queried, and then the main query table is determined more finely within the data table range. That is, by marking the level of the data table, the type it belongs to, the subject it belongs to, and the number of core business fields in the data table, the weight of each data table can be determined more comprehensively and accurately, and the one with the largest weight is then used as the main query table, so that the reference elements for determining the main query table are more comprehensive and precise, and the data query results obtained by querying the data through the main query table are more accurate, which solves the technical problem of low accuracy of data query results.

[0045] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of a data processing method based on multi-table association provided in an embodiment of the present application;

[0048] Figure 2 Another schematic diagram of a data processing method based on multi-table association provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of a data processing device based on multi-table association provided in an embodiment of the present application;

[0050] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0052] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0053] At present, the accuracy of data query results is low. Based on this, the embodiment of the present application provides a data processing method and device based on multi-table association, which can solve the technical problem of low accuracy of data query results.

[0054] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0055] Figure 1 A flowchart of a data processing method based on multi-table association provided in an embodiment of the present application. Figure 1As shown, the method includes:

[0056] Step S110, obtaining the query content input by the user.

[0057] In an optional implementation, the query content input by the user may be a query request for the data warehouse, that is, the query content input by the user may be queried in multiple data tables in the data warehouse.

[0058] As an example, user input is first received, that is, the user's query content is received through a user interface (such as a command line, a web form, a mobile application, etc.). Then, the query content is preprocessed, for example, irrelevant characters and punctuation marks are removed during noise removal, the query content is decomposed into words or phrases during word segmentation, and converted to lowercase and stemmed during standardization. Based on the preprocessed query content, the user's specific query intent can be analyzed to determine the user's specific needs, such as whether to find specific information or obtain all documents related to a certain topic.

[0059] Step S120, performing semantic recognition based on the query content to obtain a semantic recognition result, and determining a range of data tables to be queried using the association relationship between multiple data tables according to the semantic recognition result.

[0060] As a possible implementation method, in a data query scenario, multi-table association can be achieved through an association algorithm that combines source data. Specifically, semantic recognition is performed first, and then a data table range is retrieved based on the semantic recognition result.

[0061] For semantic recognition, NLP technology can be used to parse the query content, understand its intent and extract key information. After that, a query model can be built, that is, the result of semantic recognition is converted into a form that can be used for subsequent processing. Then, the relevant data table is determined, that is, the range of data tables to be queried is determined based on the association between the query model and the data table. Then, an SQL query is generated, that is, an appropriate SQL query statement is generated to retrieve relevant information from the determined data table. In the process of executing the query and returning the result, the generated SQL query is executed and the result is returned to the user.

[0062] For example, suppose a database contains the following tables: users: stores user information; orders: stores order information; products: stores product information. These tables have the following associations: users and orders are associated through user_id; orders and products are associated through product_id. The system can use Python and some commonly used NLP libraries (such as NLTK and spaCy) for semantic recognition, and use SQLAlchemy to generate and execute SQL queries.

[0063] Step S130, for each target data table in the data table range, determine the target level, target type, target subject and number of core business fields of the target data table.

[0064] Among them, the level includes the number of levels from low to high; the type includes fact table class, dimension table class, temporary table class and intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, and the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class; the subject includes business subject and non-business subject, and the subject value of the business subject is greater than the subject value of the non-business subject.

[0065] For the management of the master data table, that is, the clear rules of the master data table, for example, the rules of data table weight include: a. the level (the lower the level, the lower the weight), b. the type (fact table > dimension table > temporary table > intermediate table weight), c. the subject (business subject is higher than non-business subject), d. the number of core business fields.

[0066] As an optional implementation, data table metadata analysis is first performed, which requires obtaining and parsing the metadata information of the target data table, which usually includes but is not limited to: table name, field list and data type of each field, table comments, association relationships (foreign keys, etc.).

[0067] Then, determine the target level, where the target level refers to the hierarchical position of the data table in the entire enterprise information system or specific application field. For example, in a multi-layer enterprise architecture, there may be a basic data layer, a business processing layer, a decision support layer, etc. In this step, the system can infer the hierarchy by looking at the dependencies between data tables. For example, a table referenced as a foreign key by multiple other tables may be located in a lower basic data layer, and its expression formula is: Level = f ( Dependencies ),in Dependencies Indicates the dependency between this table and other tables.

[0068] Then determine the target type. The target type refers to the classification of data tables according to their main uses, such as transaction record tables, user information tables, log tables, etc. In this step, pattern matching can be performed based on table names, field names and descriptions, or predefined classification rules can be used. The expression formula can be: Type = g ( Name , Fields , Descriptions ).

[0069] Then determine the target topic, which is to more broadly classify the data table into a specific subject domain, such as finance, human resources, marketing, etc. In this step, by comparing the table name and field content with the known subject vocabulary, or based on the knowledge input of domain experts, the expression formula can be: Theme = h ( SubjectVocabulary , TableContent ).

[0070] Finally, determine the number of core business fields. Core business fields refer to the number of fields that are critical to the business entity represented by the data table. In this step, define which fields are core fields based on business logic. For example, the order ID and customer ID in the order table are usually core fields. The expression formula is: CoreFieldsCount = count ( CoreFieldsList ),in CoreFieldsList It is a list of core fields determined after business logic judgment.

[0071] Step S140, determining the weight of the target data table according to the number of target levels, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields.

[0072] Among them, the greater the number of target levels, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields, the greater the weight of the target data table.

[0073] Exemplarily, the type coefficient corresponding to the intermediate table class is zero (that is, the weight of the intermediate table is 0); this step may specifically include the following steps:

[0074] The weight of the target data table is determined by specifying a weight calculation formula according to the number of target levels, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields; the specified weight calculation formula is:

[0075] (a+c+d)*b;

[0076] Among them, a is the number of levels of the target level; c is the subject value of the target subject; d is the number of core business fields; and b is the type coefficient corresponding to the target type.

[0077] In an optional implementation, parameters are first defined, such as the target level (L) indicating the hierarchical position of the data table in the entire information system; the target subject (T) indicating the subject area to which the data table belongs; the target type (Ty) indicating the main usage classification of the data table; and the number of core business fields (F) indicating the number of fields that are critical to the business entity represented by the data table.

[0078] Then, assign weight coefficients to each parameter, which reflect the degree of influence of different factors on the importance of the data table. For example, the layer weight coefficient WL ; Topic weight coefficient WT ; Type weight coefficient Wt ; Weight coefficient of core business fields WF .

[0079] Then determine the scoring criteria and set specific scoring rules or value ranges for each parameter. For example, level (L): can be set as basic level (1), middle level (2), top level (3), etc.; topic (T): score each topic according to the importance of the enterprise, such as finance (5), human resources (4), marketing (3), etc.; type (Ty): score according to the importance and frequency of use of the data table, such as transaction record table (5), user information table (4), log table (3), etc. Core business fields (F): the number of fields is directly used as the score, or the level is divided according to the number of fields.

[0080] Then calculate the weight. Combining the above definition and scoring criteria, the following formula can be used to calculate the weight W of each data table: W =( L × WL )+( T × WT )+( Ty × Wt )+( F × WF ) 。 For example, assume the following parameters and weight coefficients: L : base layer (1), middle layer (2), top layer (3); theme T :Finance (5), Human Resources (4), Marketing (3); Type Ty : Transaction record table (5), user information table (4), log table (3); core business fields F : The number of fields. Assume the weight coefficient is: WL =0.4 , WT =0.3, Wt =0.2, WF =0.1. For a data table located in the middle layer, belonging to the financial subject, a transaction record table, and containing 10 core business fields, its weight is calculated as follows:

[0081] W =(2×0.4)+(5×0.3)+(5×0.2)+(10×0.1); W =0.8+1.5+1.0+1.0; W =4.3.

[0082] Finally, normalization is performed. If the weights need to be normalized to a specific range (for example, between 0 and 1), the minimum-maximum normalization method can be used:

[0083] W ′= (W - Wmin) / ( Wmax - Wmin) ;in Wmin and Wmax are the minimum and maximum values ​​among all the data table weights.

[0084] Step S150 , based on the multiple weights corresponding to the multiple target data tables, select a first target data table with the largest weight value from the multiple target data tables, and determine the first target data table as the main query table.

[0085] As an optional implementation, the main table can be: id = character table.id; as another optional implementation, the main table can also be: id = character table.id and main table; ID card = character table and ID card.

[0086] Exemplarily, the weights of all target data tables can be collected first, that is, ensure that the weight of each target data table has been calculated according to the method discussed above. Assume that there are the following data tables and their weights: Data Table A: Weight = 4.3; Data Table B: Weight = 3.8; Data Table C: Weight = 5.2; Data Table D: Weight = 3.5. Then compare the weights and determine the maximum value, and find the one with the largest value from these weights. In this example, Data Table C has the highest weight (5.2). Then select the data table with the largest weight as the main query table, and determine the data table with the highest weight as the main query table. In this example, Data Table C is selected as the main query table.

[0087] Then, record and confirm, that is, record the results of the selection and confirm whether the data table meets the business requirements and other constraints. If necessary, further verification or adjustment can be performed. After verification and confirmation, in actual applications, some additional verification steps may be required to ensure that the selected data table is indeed the most appropriate main query table. For example, data integrity check: ensure that the data of the selected data table is complete and has no serious quality problems; performance evaluation: evaluate the query performance of the selected data table to ensure that it can meet the response time requirements of the system; business logic verification: confirm with the business team whether the selected data table meets the business logic and requirements. Finally, document the selection process and results for subsequent maintenance and review. The document may include: the weight calculation method and results of each data table; the specific criteria and reasons for selecting the main query table; any additional verification and test results.

[0088] Step S160, performing data query through the main query table according to the semantic recognition result to obtain data query results corresponding to the query content.

[0089] In actual applications, after performing data query through the main query table according to the semantic recognition result, the data query result can be returned.

[0090] In an embodiment of the present application, semantic recognition is used to first roughly determine the range of the data table to be queried, and then the main query table is more finely determined within the data table range, that is, by marking the level of the data table, the type to which it belongs, the subject to which it belongs, and the number of core business fields in the data table, the weight of each data table can be determined more comprehensively and accurately, and the table with the largest weight is then used as the main query table, so that the reference elements for determining the main query table are more comprehensive and precise, and the data query results obtained by querying the data through the main query table are more accurate.

[0091] The above steps are described in detail below.

[0092] In some embodiments, Figure 2 As shown, the above step S150 may specifically include the following steps:

[0093] Step S210, based on multiple weights corresponding to multiple target data tables, selecting a first target data table having the largest weight value from the multiple target data tables; the first target data table includes a first target field name;

[0094] Step S220: if the values ​​of the multiple weights corresponding to the multiple first target data tables are the same, then the first target field name corresponding to each first target data table in the multiple first target data tables is respectively input into the natural language vectorization model to obtain the data table vector value corresponding to each first target data table;

[0095] Step S230, inputting the semantic recognition result into a natural language vectorization model to obtain a query vector value corresponding to the query content;

[0096] Step S240: Based on the multiple data table vector values ​​corresponding to the multiple first target data tables, determine the first target data table in the multiple first target data tables whose data table vector value is closest to the query vector value as the main query table.

[0097] Exemplarily, when at least two data tables have the same corresponding weights, the primary data table can be determined based on the matching relationship between the table name and the natural language vectorization. Exemplarily, for two data tables with the same weights, the field name of each data table is converted into text and then input into the model for vectorization processing to obtain vector 1, and the natural language corresponding to the semantic recognition result is input into the model for vectorization processing to obtain vector 2. For these two data tables, the data table with vector 1 closest to vector 2 is determined as the primary data table.

[0098] In an embodiment of the present application, when at least two data tables have the same corresponding weights, the main data table can be determined more accurately by performing natural language vectorization processing on the field names and semantic recognition results of the data tables, and confirming the matching relationship of the vectorization results to a close degree, thereby avoiding the situation where the same weights affect the determination process of the main data table.

[0099] In some embodiments, each data table corresponds to a table service name, and the data table includes field names and table contents; the association relationship between multiple data tables includes: a first mapping relationship between the table service name and the table content, and a second mapping relationship between the field name and the table content; the above step S120 may specifically include the following steps:

[0100] Step S122, determining the target table content related to the semantic content of the semantic recognition result according to the semantic recognition result;

[0101] Step S124, using the first mapping relationship to determine the target table business name corresponding to the target table content, and using the second mapping relationship to determine the target field name corresponding to the target table content;

[0102] Step S126, based on the target table business name and the target field name, the target table primary key is inferred and verified through data sampling, and the union of multiple target table primary keys is associated with multiple target data tables;

[0103] Step S128, determining a data table range to be queried according to multiple target data tables.

[0104] For the metadata definition rules of the mapping table (i.e., map table), for example, business name->English, a mapping relationship is determined between the table name and the content, and there is also field name->content mapping, that is, a mapping relationship is determined between the field name and the content.

[0105] In an embodiment of the present application, through data table association rules, that is, through the two mapping relationships between the above-mentioned table name and field name and the table content respectively, and through data sampling to verify the inference table primary key, the range of multiple target data tables associated with the union of the multiple table primary keys, that is, the range of data tables to be queried, can make the determined range of data tables to be queried more accurate.

[0106] In some embodiments, before the step of determining the target field name corresponding to the target table content by using the second mapping relationship, the method may further include the following steps:

[0107] Step a), inputting the semantic recognition result into a specified neural network model, so as to extract and output relevant data tables, query columns and screening conditions associated with the semantic recognition result through the specified neural network model;

[0108] Step b) adjusts the second mapping relationship based on the relevant data tables, query columns and screening conditions to optimize the association relationship between multiple data tables by specifying a neural network model.

[0109] Exemplarily, the mapping relationship between the field name and the content used for subsequent association between multiple tables may not be fixed, and the mapping relationship may be adjustable. For example, in combination with the neural network model, possible data tables, query columns, and screening conditions are extracted, that is, the input parameter is a human statement, and the output parameter is a data table, query column, and screening condition. The second mapping relationship can be adjusted according to these data tables, query columns, and screening conditions, so that the second mapping relationship between the field name and the table content is more accurate and suitable for the content to be queried.

[0110] In some embodiments, the above step S160 may specifically include the following steps:

[0111] Step S162, performing data query through the main query table, query columns and screening conditions according to the semantic recognition result to obtain SQL statement query results;

[0112] Step S164, performing SQL statement verification on the SQL statement query result to obtain the SQL statement verification result, and obtaining the natural language query result corresponding to the query content based on the SQL statement verification result.

[0113] As an example, for writing SQL statements, you can combine the above query columns and filter conditions and write SQL through the model, that is, the input parameter is: natural language generated by the program, and the output parameter is the SQL statement. Then, you can verify the SQL statement to obtain a more accurate final query result.

[0114] In some embodiments, before the above step of using the first mapping relationship to determine the target table business name corresponding to the target table content, the method may also include the following steps: determining the target table business name according to a naming rule that combines data warehouse keywords, business primary keys and time keywords.

[0115] In practical applications, the above data tables can be named according to certain naming rules. For example, the target table business name can be named in the form of "data warehouse keyword + business primary key + time keyword" to make the target table business name more comprehensive, so as to facilitate the efficient query process of the data table and improve the query efficiency of the data table.

[0116] Figure 3 A schematic diagram of the structure of a data processing device based on multi-table association is provided. Figure 3 As shown, the data processing device 300 based on multi-table association includes:

[0117] The acquisition module 301 is used to acquire the query content input by the user;

[0118] The recognition module 302 is used to perform semantic recognition based on the query content to obtain a semantic recognition result, and determine the range of the data table to be queried by using the association relationship between multiple data tables according to the semantic recognition result;

[0119] The first determination module 303 is used to determine, for each target data table in the data table range, the target level, the target type, the target subject and the number of core business fields in the target data table; wherein the level includes the number of levels from low to high; the type includes the fact table class, the dimension table class, the temporary table class and the intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, and the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class; the subject includes business subjects and non-business subjects, and the subject value of the business subject is greater than the subject value of the non-business subject;

[0120] The second determination module 304 is used to determine the weight of the target data table according to the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields; wherein, the greater the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields, the greater the weight of the target data table;

[0121] A selection module 305 is configured to select a first target data table having the largest weight value from the multiple target data tables based on the multiple weights corresponding to the multiple target data tables, and determine the first target data table as a main query table;

[0122] The query module 306 is used to perform data query through the main query table according to the semantic recognition result to obtain the data query result corresponding to the query content.

[0123] In some embodiments, the type coefficient corresponding to the intermediate table class is zero; the second determination module is specifically used for:

[0124] The weight of the target data table is determined by a specified weight calculation formula according to the number of levels of the target level, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields; the specified weight calculation formula is:

[0125] (a+c+d)*b;

[0126] Among them, a is the level number of the target level; c is the topic value of the target topic; d is the number of core business fields; and b is the type coefficient corresponding to the target type.

[0127] In some embodiments, the selection module is specifically configured to:

[0128] Based on the multiple weights corresponding to the multiple target data tables, selecting a first target data table having the largest value of the weight from the multiple target data tables; the first target data table includes a first target field name;

[0129] If the values ​​of the multiple weights corresponding to the multiple first target data tables are the same, the first target field name corresponding to each of the multiple first target data tables is respectively input into the natural language vectorization model to obtain the data table vector value corresponding to each of the first target data tables;

[0130] Inputting the semantic recognition result into a natural language vectorization model to obtain a query vector value corresponding to the query content;

[0131] Based on the multiple data table vector values ​​corresponding to the multiple first target data tables, a first target data table among the multiple first target data tables whose data table vector value is closest to the query vector value is determined as a main query table.

[0132] In some embodiments, each of the data tables corresponds to a table service name, and the data table includes a field name and a table content; the association relationship between the multiple data tables includes: a first mapping relationship between the table service name and the table content, and a second mapping relationship between the field name and the table content; the identification module is specifically used to:

[0133] Determining target table content related to the semantic content of the semantic recognition result according to the semantic recognition result;

[0134] Determine the target table service name corresponding to the target table content by using the first mapping relationship, and determine the target field name corresponding to the target table content by using the second mapping relationship;

[0135] Based on the target table business name and the target field name, the target table primary key is inferred and verified through data sampling, and the union of multiple target table primary keys is associated with multiple target data tables;

[0136] The data table range to be queried is determined according to the multiple target data tables.

[0137] In some embodiments, the device also includes an adjustment module for: before using the second mapping relationship to determine the target field name corresponding to the target table content, inputting the semantic recognition result into a specified neural network model, so as to extract and output relevant data tables, query columns and filtering conditions associated with the semantic recognition result through the specified neural network model; adjusting the second mapping relationship based on the relevant data tables, the query columns and the filtering conditions, so as to optimize the association relationship between the multiple data tables through the specified neural network model.

[0138] In some embodiments, the query module is specifically used to:

[0139] According to the semantic recognition result, data query is performed through the main query table, the query column and the screening condition to obtain an SQL statement query result;

[0140] Performing SQL statement verification on the SQL statement query result to obtain an SQL statement verification result, and obtaining a natural language query result corresponding to the query content based on the SQL statement verification result.

[0141] In some embodiments, the apparatus further comprises:

[0142] The third determination module is used to determine the target table business name according to a naming rule combining a data warehouse keyword, a business primary key and a time keyword before determining the target table business name corresponding to the target table content by using the first mapping relationship.

[0143] The data processing device based on multi-table association provided in the embodiment of the present application has the same technical features as the data processing method based on multi-table association provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0144] An electronic device provided in an embodiment of the present application is Figure 4As shown, the electronic device 400 includes a processor 402 and a memory 401, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method provided in the above embodiment when executing the computer program.

[0145] See also Figure 4 The electronic device further includes: a bus 403 and a communication interface 404, a processor 402, a communication interface 404 and a memory 401 are connected via the bus 403; the processor 402 is used to execute an executable module stored in the memory 401, such as a computer program.

[0146] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 404 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0147] The bus 403 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0148] Among them, the memory 401 is used to store programs, and the processor 402 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present application can be applied to the processor 402 or implemented by the processor 402.

[0149] The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 402. The above processor 402 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor are combined to execute. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 402 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0150] Corresponding to the above-mentioned data processing method based on multi-table association, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned data processing method based on multi-table association.

[0151] The data processing device based on multi-table association provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present application, its implementation principle and the technical effect produced are the same as those in the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0152] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0153] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the data processing method based on multi-table association described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0157] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0158] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data processing method based on multi-table association, characterized in that: The method comprises: Get the query content entered by the user; Performing semantic recognition based on the query content to obtain a semantic recognition result, and determining a range of data tables to be queried using an association relationship between multiple data tables according to the semantic recognition result; For each target data table in the data table range, determine the target level, target type, target subject and the number of core business fields in the target data table; wherein the level includes the number of levels from low to high; the type includes the fact table class, the dimension table class, the temporary table class and the intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, and the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class; the subject includes business subjects and non-business subjects, and the subject value of the business subject is greater than the subject value of the non-business subject; The weight of the target data table is determined according to the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields; wherein the greater the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields, the greater the weight of the target data table; Based on the multiple weights corresponding to the multiple target data tables, selecting a first target data table having the largest value of the weight from the multiple target data tables, and determining the first target data table as a main query table; Performing data query through the main query table according to the semantic recognition result to obtain data query results corresponding to the query content; Each of the data tables corresponds to a table service name, and the data table includes a field name and table content; the association relationship between the multiple data tables includes: a first mapping relationship between the table service name and the table content, and a second mapping relationship between the field name and the table content; The step of determining the range of the data tables to be queried by utilizing the association relationship between the multiple data tables according to the semantic recognition result comprises: Determining target table content related to the semantic content of the semantic recognition result according to the semantic recognition result; Determine the target table service name corresponding to the target table content by using the first mapping relationship, and determine the target field name corresponding to the target table content by using the second mapping relationship; Based on the target table business name and the target field name, the target table primary key is inferred and verified through data sampling, and the union of multiple target table primary keys is associated with multiple target data tables; The data table range to be queried is determined according to the multiple target data tables.

2. The method according to claim 1, characterized in that The type coefficient corresponding to the intermediate table class is zero; the step of determining the weight of the target data table according to the number of levels of the target level, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields includes: The weight of the target data table is determined by a specified weight calculation formula according to the number of levels of the target level, the subject value of the target subject, the type coefficient corresponding to the target type, and the number of core business fields; the specified weight calculation formula is: (a+c+d)*b; Among them, a is the level number of the target level; c is the topic value of the target topic; d is the number of core business fields; and b is the type coefficient corresponding to the target type.

3. The method according to claim 2, characterized in that The step of selecting a first target data table having the largest weight value from the multiple target data tables based on the multiple weights corresponding to the multiple target data tables, and determining the first target data table as the main query table comprises: Based on the multiple weights corresponding to the multiple target data tables, selecting a first target data table having the largest value of the weight from the multiple target data tables; the first target data table includes a first target field name; If the values ​​of the multiple weights corresponding to the multiple first target data tables are the same, the first target field name corresponding to each of the multiple first target data tables is respectively input into the natural language vectorization model to obtain the data table vector value corresponding to each of the first target data tables; Inputting the semantic recognition result into a natural language vectorization model to obtain a query vector value corresponding to the query content; Based on the multiple data table vector values ​​corresponding to the multiple first target data tables, a first target data table among the multiple first target data tables whose data table vector value is closest to the query vector value is determined as a main query table.

4. The method according to claim 1, characterized in that: Before the step of determining the target field name corresponding to the target table content by using the second mapping relationship, the method further includes: Inputting the semantic recognition result into a specified neural network model, so as to extract and output relevant data tables, query columns and screening conditions associated with the semantic recognition result through the specified neural network model; The second mapping relationship is adjusted based on the related data table, the query column and the screening condition to optimize the association relationship between the multiple data tables through the specified neural network model.

5. The method according to claim 4, characterized in that The step of performing data query through the main query table according to the semantic recognition result to obtain the data query result corresponding to the query content includes: According to the semantic recognition result, data query is performed through the main query table, the query column and the screening condition to obtain an SQL statement query result; Performing SQL statement verification on the SQL statement query result to obtain an SQL statement verification result, and obtaining a natural language query result corresponding to the query content based on the SQL statement verification result.

6. The method according to claim 1, characterized in that Before the step of determining the target table service name corresponding to the target table content by using the first mapping relationship, the method further includes: The target table business name is determined according to a naming rule combining a data warehouse keyword, a business primary key, and a time keyword.

7. A data processing device based on multi-table association, characterized in that: include: The acquisition module is used to obtain the query content input by the user; An identification module, used for performing semantic identification based on the query content to obtain a semantic identification result, and determining a range of data tables to be queried by using an association relationship between multiple data tables according to the semantic identification result; The first determination module is used to determine, for each target data table in the data table range, the target level, the target type, the target subject and the number of core business fields in the target data table; wherein the level includes the number of levels from low to high; the type includes the fact table class, the dimension table class, the temporary table class and the intermediate table class, the type coefficient corresponding to the fact table class is greater than the type coefficient corresponding to the dimension table class, the type coefficient corresponding to the dimension table class is greater than the type coefficient corresponding to the temporary table class, and the type coefficient corresponding to the temporary table class is greater than the type coefficient corresponding to the intermediate table class; the subject includes business subjects and non-business subjects, and the subject value of the business subject is greater than the subject value of the non-business subject; A second determination module is used to determine the weight of the target data table according to the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields; wherein, the greater the number of levels of the target level, the theme value of the target theme, the type coefficient corresponding to the target type, and the number of core business fields, the greater the weight of the target data table; A selection module, configured to select a first target data table having the largest value of the weight from the multiple target data tables based on the multiple weights corresponding to the multiple target data tables, and determine the first target data table as a main query table; A query module, used to perform data query through the main query table according to the semantic recognition result, and obtain a data query result corresponding to the query content; Each of the data tables corresponds to a table service name, and the data table includes a field name and table content; the association relationship between the multiple data tables includes: a first mapping relationship between the table service name and the table content, and a second mapping relationship between the field name and the table content; the identification module is specifically used for: Determining target table content related to the semantic content of the semantic recognition result according to the semantic recognition result; Determine the target table service name corresponding to the target table content by using the first mapping relationship, and determine the target field name corresponding to the target table content by using the second mapping relationship; Based on the target table business name and the target field name, the target table primary key is inferred and verified through data sampling, and the union of multiple target table primary keys is associated with multiple target data tables; The data table range to be queried is determined according to the multiple target data tables.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 6.

Images