Data preparation method, system, and device for AIGC-based interaction analysis, and medium

Abstract

The present application relates to the technical field of electric digital data processing, and provides a data preparation method, system, and device for AIGC-based interaction analysis, and a medium. The method comprises: by means of a pre-trained text label recognition model, determining a label sequence corresponding to an interaction statement having experienced first word segmentation processing; on the basis of a preset expression construction rule, determining a label combination sequence corresponding to the label sequence and a conditional expression corresponding to the label combination sequence; by means of a preconfigured database candidate text dictionary and a preset matching rule, determining a strictly matched keyword and a fuzzily matched keyword corresponding to the interaction statement having experienced second word segmentation processing; on the basis of the strictly matched keyword and a similar candidate text corresponding to the fuzzily matched keyword, determining a word segmentation result corresponding to the second word segmentation processing; on the basis of a plurality of preset SQL statement rule matching templates, generating an interaction SQL query statement, so as to determine, on the basis of the interaction SQL query statement, interaction response information corresponding to the interaction statement.

Description

A data preparation method, system, device and medium for AIGC interactive analysis

[0001] The present application claims priority to the Chinese patent application No. 202410516640.1, filed on April 28, 2024, and entitled "A data preparation method, system, device and medium for AIGC interactive analysis", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of electronic digital data processing, and in particular to a data preparation method, system, device and medium for AIGC interactive analysis. BACKGROUND

[0003] Artificial Intelligence Generated Content (AIGC) refers to a technology that generates relevant content based on the learning and recognition of existing data through appropriate generalization ability, by using artificial intelligence technologies such as generative adversarial networks and large pre-trained models.

[0004] At present, AIGC can recognize and understand natural language questions of users in dialog-based BI products such as ChatBI, and achieve interaction with users. However, the understanding ability of the current dialog-based BI products for user inquiry questions is still very limited.

[0005] When a dialog-based BI product understands natural language, it usually includes the process of tokenizing natural language and matching the keywords extracted by tokenization with fields in a database. For tokenization, the industry mostly uses four ways such as rule-based matching tokenization, statistical tokenization, deep learning tokenization, and large model fine-tuning tokenization. In actual use, the inventors found that the tokenization results obtained by these four ways have problems such as language recognition ambiguity, inability to be directly applied to converting structured query language SQL, and inability to accurately generate SQL that implements complex logic, which makes the dialog-based BI product still at a low level of intelligence.

[0006] SUMMARY

[0007] To solve the above problems, the present application provides a data preparation method, system, device and medium for AIGC interactive analysis.

[0008] In one aspect, the present application provides a data preparation method for AIGC interactive analysis, which comprises:

[0009] determine a label sequence corresponding to the interactive sentence after first word segmentation by using a pre-trained text label recognition model; the label sequence includes preset labels corresponding to each first text word segmentation and arranged in the order of the interactive sentence;

[0010] determine a label combination sequence corresponding to the label sequence and a conditional expression corresponding to the label combination sequence based on a preset expression construction rule; the preset expression construction rule is used to combine the preset labels in the label sequence and obtain the label combination sequence; each label combination sequence has a unique corresponding conditional expression;

[0011] determine strictly matched keywords and fuzzy matched keywords corresponding to the interactive sentence after second word segmentation by using a pre-configured database candidate text dictionary and a preset matching rule; the preset matching rule is used to match candidate texts in the database candidate text dictionary that are consistent with the second text word segmentation;

[0012] determine a word segmentation result corresponding to the second word segmentation based on the strictly matched keywords and similar candidate texts corresponding to the fuzzy matched keywords;

[0013] generate an interactive SQL query statement corresponding to the conditional expression and the word segmentation result based on a plurality of preset SQL statement rule matching templates, and determine an interactive response information corresponding to the interactive sentence based on the interactive SQL query statement.

[0014] In an implementation manner of the present application, before determining a label sequence corresponding to the interactive sentence after first word segmentation by using a pre-trained text label recognition model, the method further includes:

[0015] obtain a pre-configured text normalization mapping table;

[0016] after receiving the interactive sentence from the user terminal, determine a text to be normalized in the interactive sentence by using the text normalization mapping table, and map the text format of the text to be normalized to a normalized format to obtain the interactive sentence in the normalized format; the normalized format at least includes a number normalization format, a time normalization format, a punctuation normalization format, and a reserved word normalization format.

[0017] In an implementation manner of the present application, before determining a label sequence corresponding to the interactive sentence after first word segmentation by using a pre-trained text label recognition model, the method further includes:

[0018] obtaining a pre-constructed sentence training sample set, wherein the sentence training sample set at least includes a plurality of interactive sentence samples and corresponding SQL query statements, and each of the interactive sentence samples is pre-labeled with the preset label used to compose the conditional expression;

[0019] inputting each of the interactive sentence samples in the sentence training sample set into a text label recognition model to be trained, so that the text label recognition model performs the first word segmentation processing on each of the interactive sentence samples through a BERT model thereof, and outputs a sample predicted label sequence corresponding to each of the interactive sentence samples after the first word segmentation processing through an output layer;

[0020] determining a difference degree value of each of the sample predicted label sequences and a corresponding real label sequence through a cross-entropy loss function, until the difference degree value is less than a preset threshold, and obtaining the trained text label recognition model.

[0021] In an implementation manner of the present application, the strictly matched keywords and the fuzzy matched keywords corresponding to the interactive sentence after the second word segmentation processing are determined through a pre-set database candidate text dictionary and a preset matching rule, specifically including:

[0022] performing the second word segmentation processing on the interactive sentence through a HanLP word segmenter, and identifying the second text segmentation consistent with the candidate text through the database candidate text dictionary;

[0023] performing part-of-speech tagging on the second text segmentation consistent with the candidate text according to the pre-labeled part-of-speech corresponding to the candidate text, to obtain the strictly matched keywords;

[0024] taking the second text segmentation inconsistent with any of the candidate texts as the fuzzy matched keywords.

[0025] In an implementation manner of the present application, the word segmentation result corresponding to the second word segmentation processing is determined based on the strictly matched keywords and the similar candidate texts corresponding to the fuzzy matched keywords, specifically including:

[0026] converting the fuzzy matched keywords into keyword vectors through a pre-trained word vector model;

[0027] calculating text similarities between the keyword vectors and candidate text vectors corresponding to each of the candidate texts, and inputting each of the text similarities into a preset hierarchical clustering algorithm as an input value, to determine a similar candidate text set corresponding to the fuzzy matched keywords;

[0028] The similar candidate text set and the fuzzy matching keyword are matched according to a preset screening rule, to obtain each candidate text pair corresponding to the fuzzy matching keyword.

[0029] The second word segmentation result of the interactive sentence is determined according to the similar candidate text, the corresponding pre-labeled part-of-speech, and the strictly matching keyword. The second word segmentation result includes a query candidate text and a part-of-speech thereof. The part-of-speech at least includes a field and a value.

[0030] In an implementation manner of the present application, the preset hierarchical clustering algorithm is an Average-Linkage clustering algorithm.

[0031] In an implementation manner of the present application, the interactive SQL query statement corresponding to the condition expression and the word segmentation result is generated based on a plurality of preset SQL statement rule matching templates, and specifically includes:

[0032] The plurality of preset SQL statement rule matching templates are determined based on a predefined operation of a user. One of the preset SQL statement rule matching templates corresponds to one database query type. The preset SQL statement rule matching template includes a to-be-filled slot for constructing a database query statement.

[0033] The interactive SQL query statement is obtained by matching the corresponding preset SQL statement rule matching template according to the condition expression and the word segmentation result, and filling the corresponding to-be-filled slot. The condition expression includes a query condition corresponding to the interactive SQL query statement. The word segmentation result includes a query candidate text corresponding to the query condition and used for constructing the interactive SQL query statement.

[0034] On the other hand, the present application also provides a data preparation system for AIGC interactive analysis, which comprises:

[0035] A label sequence determination module is configured to determine a label sequence corresponding to the interactive sentence after the first word segmentation by using a pre-trained text label recognition model. The label sequence includes a preset label corresponding to each first text segmentation in the order of the interactive sentence.

[0036] A condition expression determination module is configured to determine a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence based on a preset expression construction rule. The preset expression construction rule is used to combine a plurality of preset labels in the label sequence and obtain the label combination sequence. Each label combination sequence has a unique corresponding condition expression.

[0037] The keyword determination module is configured to determine strict matching keywords and fuzzy matching keywords corresponding to the interactive sentence subjected to the second word segmentation processing by using a preset database candidate text dictionary and a preset matching rule, wherein the preset matching rule is used to match candidate texts in the database candidate text dictionary that are consistent with the second text segmentation;

[0038] The word segmentation result determination module is configured to determine a word segmentation result corresponding to the second word segmentation processing based on the strict matching keywords and similar candidate texts corresponding to the fuzzy matching keywords.

[0039] The sentence generation module is configured to generate an interactive SQL query sentence corresponding to the condition expression and the word segmentation result based on a plurality of preset SQL sentence rule matching templates, and determine interactive response information corresponding to the interactive sentence based on the interactive SQL query sentence.

[0040] In another aspect, the embodiments of the present application also provide a data preparation device for AIGC interactive analysis, the device comprising:

[0041] at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0042] determine a label sequence corresponding to the interactive sentence subjected to the first word segmentation processing by using a pre-trained text label recognition model, wherein the label sequence comprises preset labels corresponding to each first text segmentation arranged in the order of the interactive sentence;

[0043] determine a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence based on a preset expression construction rule, wherein the preset expression construction rule is used to combine a plurality of preset labels in the label sequence and obtain the label combination sequence, and each label combination sequence has a unique corresponding condition expression;

[0044] determine strict matching keywords and fuzzy matching keywords corresponding to the interactive sentence subjected to the second word segmentation processing by using a preset database candidate text dictionary and a preset matching rule, wherein the preset matching rule is used to match candidate texts in the database candidate text dictionary that are consistent with the second text segmentation;

[0045] determine a word segmentation result corresponding to the second word segmentation processing based on the strict matching keywords and similar candidate texts corresponding to the fuzzy matching keywords.

[0046] generate an interactive SQL query statement corresponding to the condition expression and the segmentation result based on a plurality of preset SQL statement rule matching templates, and determine the interactive response information corresponding to the interactive statement based on the interactive SQL query statement.

[0047] In another aspect, the embodiments of the present application also provide a non-volatile computer storage medium storing computer executable instructions, which are configured to:

[0048] determine a label sequence corresponding to the first segmented interactive statement through a pre-trained text label recognition model; wherein the label sequence includes a preset label corresponding to each first text segmentation arranged in the order of the interactive statement;

[0049] determine a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence based on a preset expression construction rule; the preset expression construction rule is used to combine a plurality of preset labels in the label sequence and obtain the label combination sequence; each label combination sequence has a unique corresponding condition expression;

[0050] determine a strictly matched keyword and a fuzzy matched keyword corresponding to the second segmented interactive statement through a preconfigured database candidate text dictionary and a preset matching rule; wherein the preset matching rule is used to match a candidate text consistent with the second text segmentation in the database candidate text dictionary;

[0051] determine a segmentation result corresponding to the second segmentation processing based on the strictly matched keyword and a similar candidate text corresponding to the fuzzy matched keyword;

[0052] generate an interactive SQL query statement corresponding to the condition expression and the segmentation result based on a plurality of preset SQL statement rule matching templates, and determine the interactive response information corresponding to the interactive statement based on the interactive SQL query statement.

[0053] Compared with the prior art, the present application has the following significant effects:

[0054] Through the above scheme, the present application can realize accurate segmentation processing of natural language and ensure that the interactive statement can be successfully matched with the database field or value. At the same time, the condition expression in the interactive statement and the corresponding database field or value are identified, which can guarantee the accuracy of the generated interactive SQL query statement, effectively convert natural language into executable SQL statements. It can also avoid ambiguity in converting SQL statements, and realize the generation of complex logic SQL statements, providing sufficient data preparation conditions for AIGC interactive analysis. BRIEF DESCRIPTION OF DRAWINGS

[0055] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0056] Figure 1 is a flowchart illustrating a data preparation method for AIGC interactive analysis in an embodiment of this application.

[0057] Figure 2 is a schematic diagram of the natural language response process corresponding to a data preparation method for AIGC interactive analysis in an embodiment of this application.

[0058] Figure 3 is a schematic diagram of the structure of a data preparation system for AIGC interactive analysis in an embodiment of this application;

[0059] Figure 4 is a schematic diagram of the structure of a data preparation device for AIGC interactive analysis in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Existing conversational BI products still have limited ability to understand user queries. The foundation for converting natural language into valid, executable SQL lies in the accurate and reasonable segmentation of the natural language, thereby successfully matching the extracted keywords with fields or values ​​in the database. There are four main natural language segmentation techniques in the industry, each with its own limitations and implementation difficulties:

[0062] 1. Rule-based word segmentation: Relies on predefined dictionaries and grammatical rules to identify and segment words in text. It is easy to implement and works well for some fixed and formatted texts, but its ability to identify new words is weak and its ability to handle ambiguity and complex sentence structures is limited.

[0063] 2. Statistical word segmentation: By statistically analyzing the co-occurrence information of words in the corpus, the boundary probability of words is calculated. It does not rely entirely on a dictionary and can identify new words to a certain extent. However, it requires a large amount of labeled data for model training, has poor adaptability to changes in language usage habits, and its word segmentation accuracy is limited by the quality and coverage of the training data.

[0064] 3. Deep learning-based word segmentation: better understanding of context and semantic information, stronger processing ability for ambiguous words, usually no need for manual feature design, the model can automatically learn features from data, but the model is strongly dependent on training data, and the quality of data directly affects the segmentation effect.

[0065] 4. Word segmentation based on large model fine-tuning: can utilize the powerful language understanding ability of large models, can more accurately handle complex text and ambiguity. The disadvantage is that it also requires a large amount of computing resources, and it is necessary to collect user questions as fine-tuning training databases and perform data labeling, the data source is less, and the domain is single, and it is difficult to obtain; the trained large model is still only effective for natural language questions in this field.

[0066] Using the above four methods alone cannot truly apply to converting SQL and eliminating question ambiguity and generating complex logic SQL.

[0067] Therefore, the embodiments of the present application provide a data preparation method and system for AIGC interactive analysis, a device and a medium, to solve the problem that the current natural language word segmentation is not accurate enough, resulting in language recognition ambiguity in the word segmentation result, which cannot be directly applied to converting SQL and cannot accurately generate complex logic SQL.

[0068] The various embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0069] The embodiments of the present application provide a data preparation method for AIGC interactive analysis, as shown in FIG. 1, which can include steps S101-S105:

[0070] S101, the server determines the label sequence corresponding to the first word segmentation processed interactive sentence through the pre-trained text label recognition model.

[0071] The label sequence includes a preset label corresponding to each first text word segmentation arranged in the order of the interactive sentence.

[0072] It should be noted that the server as the execution subject of the data preparation method for AIGC interactive analysis only exists as an example, and the execution subject is not limited to the server, and the present application does not make specific limitations thereto.

[0073] The text label recognition model can process and recognize the interactive sentence and output a label sequence corresponding to the interactive sentence. For example, the interactive sentence is "find employees with an age greater than 30 years old", the text label recognition model labels "age" as a preset label: field name (F), labels "greater than" as a preset label: operator (O), and labels "30 years old" as a preset label: value (V). The preset label can be understood as a component of the conditional expression, which can at least include: field name, operator and value, which can be set by the user according to the actual use scene, and the present application does not make specific limitation. The text label recognition model can be a BERT model (BERT-Base-Chinese pre-training model based on Transformer) added with an output layer suitable for a sequence labeling task, wherein the sequence labeling task refers to labeling the preset label for the first text segmentation output by the tokenizer of the BERT model, and the output layer can be a fully connected layer, and the output dimension thereof is equal to the number of preset labels (such as field name, operator, value) labeled in the sequence labeling task.

[0074] For the pre-training process of the above text label recognition model, the following embodiments can be referred to. Specifically, before determining the label sequence corresponding to the first segmented interactive sentence through the pre-trained text label recognition model, the method further comprises:

[0075] The server obtains a pre-constructed sentence training sample set. The sentence training sample set at least includes a plurality of interactive sentence samples and their corresponding SQL query statements. Each interactive sentence sample is pre-labeled with a preset label used to compose a conditional expression. Then, each interactive sentence sample in the sentence training sample set is input into the text label recognition model to be trained, so that the text label recognition model performs first segmentation processing on each interactive sentence sample through its BERT model, and outputs a sample predicted label sequence corresponding to the first segmented interactive sentence sample through the output layer. Then, the difference degree value of each sample predicted label sequence and the corresponding real label sequence is determined through a cross-entropy loss function until the difference degree value is less than a preset threshold, and a trained text label recognition model is obtained.

[0076] In other words, a sentence training sample set containing interactive sentence samples and their corresponding SQL query statements can be pre-constructed, and the components of the conditional expression in each interactive sentence sample, such as field name, operator and value, can be labeled. The interactive sentence in the present application can be understood as natural language input by the user into the server. The server uses the sentence training sample set to pre-train the text label recognition model.

[0077] In the pre-training process, first, the tokenizer of the BERT model is used to perform first tokenization processing on the interactive sentence sample, and the tokenized text is converted into an input format that the model can understand, such as a numerical ID sequence of words. At the same time, the label corresponding to the tokenized text is converted into a corresponding sample prediction label sequence, and the output layer can be used to output the sample prediction label sequence. In this process, the server uses a cross-entropy loss function to train the model, which calculates the difference between the sample prediction label sequence output by the model and the true label sequence, and quantifies the difference as a difference value through the function value of the cross-entropy loss function. In the case where the difference value is less than the pre-set threshold value set by the user, the text label recognition model is trained.

[0078] In another embodiment of the present application, the interactive sentence usually contains non-standard words and expressions, which will affect the subsequent tokenization and recognition understanding of the interactive sentence. Therefore, before determining the label sequence corresponding to the first tokenized interactive sentence through the pre-trained text label recognition model, it further includes:

[0079] The server obtains a pre-constructed text normalization mapping table. After receiving the interactive sentence from the user terminal, the text normalization mapping table is used to determine the text to be standardized in the interactive sentence, and the text format of the text to be standardized is mapped to a standardized format to obtain an interactive sentence in a standardized format. The standardized format at least includes a numerical standard format, a time standard format, a punctuation symbol standard format, and a reserved word standard format.

[0080] That is, in order to normalize the input interactive sentence and convert it into a form that is easier for machines to understand and process, a text normalization mapping table can be constructed in advance. Through the text normalization mapping table, after receiving the non-normalized interactive sentence, text normalization mapping can be performed first, and then first segmentation processing, second segmentation processing, etc. can be performed. Among them, text normalization mapping can map interactive sentences to digital standard format, time standard format, punctuation symbol standard format, and reserved word standard format. Among them, the number normalization, such as the normalization of "100,000 yuan" to "10000 yuan", can map Chinese numbers and quantifiers (such as "one", "ten thousand", "thousand", etc.) to the corresponding numerical values, and through the analysis and calculation of the numerical values and quantifiers in the expression, it is converted to a standard digital representation. Time normalization: such as "2023-05-01" from "2023-05-01", the normalization of time usually needs to parse the time expression in natural language, and then convert it to a standard format (such as ISO 8601). The server can also identify different date formats through regular expressions, or use special libraries (such as dateutil.parser in Python) to parse date expressions in natural language. Among them, for relative time expressions (such as "next year", "next month", etc.), the actual date can be calculated in combination with the current date context. For label symbol standard format, it can include punctuation symbol cleaning; the reserved word standard format can use part-of-speech tagging technology to remove mood words and virtual words, such as "please help me query", "huh", "which" and other word texts.

[0081] Through the above text normalization processing, it is easier to segment the normalized text, which can more efficiently and accurately convert natural language into SQL statements, and improve the operation speed and intelligent level of the conversational BI product.

[0082] S102, the server determines a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence based on a preset expression construction rule.

[0083] The above-mentioned preset expression construction rule is used to combine multiple preset labels in the label sequence and obtain a label combination sequence. Each label combination sequence has a unique corresponding condition expression.

[0084] In the embodiment of the present application, the server can be pre-provided with a preset expression construction rule, which contains an association relationship between a label combination sequence obtained by combining different preset labels and different conditional expressions. For example, after a sentence of interactive statement is processed by a text label identification model, the output label sequence is a label sequence [field name, operator, value] composed of a field name, an operator and a value that are successively labeled. Through the preset expression construction rule, it can be determined that the label sequence has a unique corresponding conditional expression, i.e., the label sequence can constitute a conditional expression, which is used for subsequent conversion into SQL to query the content to be queried corresponding to the conditional expression. For the conditional expression, a more intuitive understanding is, for example, age-greater than-30 years old, which is a conditional expression.

[0085] The above scheme of the present application trains a model based on deep learning, and by using the trained text label identification model, a conditional expression contained in an interactive statement can be accurately and efficiently obtained, thereby providing support for subsequent generation of an accurate interactive SQL query statement.

[0086] In S103, the server determines the strictly matched keywords and the fuzzy matched keywords corresponding to the second processed interactive statement by using the pre-provided database candidate text dictionary and the preset matching rule.

[0087] The preset matching rule is used to match the candidate text in the database candidate text dictionary that is consistent with the second text segmentation.

[0088] The pre-provided database candidate text dictionary can be constructed by all table names, field names and values in the database. The server can traverse all tables in the database, extract table names, field names and cell values of non-numeric types, and perform a de-duplication process on the cell values. The server can arrange all collected information in the format required by the HanLP segmenter, including each word and its part of speech (for the content in the database, the part of speech can be customized, such as table for table name, field for field name, and value for value), thereby obtaining a database candidate text dictionary containing a plurality of candidate texts. The candidate text can be a field or a value, which is not limited in the present application.

[0089] After the database candidate text dictionary is pre-provided, in the embodiment of the present application, the strictly matched keywords and the fuzzy matched keywords corresponding to the second processed interactive statement are determined by using the pre-provided database candidate text dictionary and the preset matching rule, which specifically includes:

[0090] The server performs second word segmentation processing on the interactive sentence through a HanLP word segmenter, and identifies second text segmentation consistent with the candidate text through a database candidate text dictionary. According to the pre-labeled part-of-speech corresponding to the candidate text, the part-of-speech of the second text segmentation consistent with the candidate text is labeled to obtain strictly matched keywords. The second text segmentation inconsistent with any candidate text is taken as a fuzzy matched keyword.

[0091] That is, the server can perform second word segmentation processing on the interactive sentence through the HanLP word segmenter of the Chinese language processing package tool. The second word segmentation processing can be performed on the interactive sentence after text normalization processing. During the second word segmentation processing, the word segmenter can use the database candidate text dictionary to identify the second text segmentation consistent with the candidate text, and label the part-of-speech of the second text segmentation, so as to obtain strictly matched keywords. The second text segmentation that is not successfully matched with the candidate text in the database candidate text dictionary is taken as a fuzzy matched keyword.

[0092] For example, the interactive sentence without text normalization processing is "What is the total transaction volume of stocks in the first industry of non-bank financial industry?"; the database candidate text dictionary includes "first industry", "non-bank financial", "stock", and "transaction volume". Among them, "first industry", "stock", and "transaction volume" in the interactive sentence can be recognized and extracted by the HanLP word segmenter, and are labeled as "fields", that is, strictly matched keywords; but "non-bank financial" does not have consistent candidate text in the database candidate text dictionary (not completely consistent with "non-bank financial"), and will be returned as a fuzzy matched keyword.

[0093] In S104, the server determines the word segmentation result corresponding to the second word segmentation processing based on the strictly matched keywords and the similar candidate text corresponding to the fuzzy matched keyword.

[0094] In the embodiments of the present application, the word segmentation result corresponding to the second word segmentation processing is determined based on the strictly matched keywords and the similar candidate text corresponding to the fuzzy matched keyword, which specifically includes:

[0095] First, the fuzzy matched keyword is processed to determine the similar candidate text corresponding thereto. Specifically, the fuzzy matched keyword can be converted into a keyword vector through a pre-trained word vector model. The text similarity between the keyword vector and the candidate text vector corresponding to each candidate text is calculated, and each text similarity is input into a pre-set hierarchical clustering algorithm as an input value to determine a similar candidate text set corresponding to the fuzzy matched keyword. Through a pre-set screening rule, each candidate text pair generated by the similar candidate text set and the fuzzy matched keyword is screened to obtain the similar candidate text corresponding to the fuzzy matched keyword.

[0096] Wherein, in determining the similar candidate text, the server can use a word vector model such as Word2Vec, GloVe, BERT, etc. to convert the fuzzy matching keyword into a vector representation to obtain a keyword vector; then the text similarity between the keyword vector and the candidate text vector of each candidate text in the database candidate text dictionary can be calculated. The text similarity can be cosine similarity, the formula is as follows:

[0097] Wherein, cos t is the cosine similarity of the keyword vector A and the candidate text vector B, A x represents the component of the keyword vector A, B x the component of the candidate text vector, and y is the total amount of vector components.

[0098] After obtaining the text similarity, the server can use it as input to use a hierarchical clustering algorithm for clustering analysis. In the embodiments of the present application, the hierarchical clustering algorithm can be an Average-Linkage clustering algorithm. In the clustering analysis, the average distance of all possible point pairs between clusters can be calculated, and the two clusters with the smallest distance are selected for merging. In this process, the candidate texts and the fuzzy matching keywords can be grouped according to the level of text similarity to obtain a combination of words close in semantics to form a candidate text pair, and form a similar candidate text set.

[0099] Further, the server can apply record linking technology to identify and link the best matching items in the similar candidate text set. Specifically, by setting a similarity threshold, candidate text pairs exceeding the threshold are considered to be effective matches, and one or more candidate text pairs are selected. Then a preset filtering rule can be used to obtain a candidate text pair, that is, to determine the similar candidate text that is the most matched candidate text with the fuzzy matching keyword. The preset filtering rule can be to select the matching item with the highest text similarity, to solve the problem in a rule-based manner in combination with domain knowledge, etc. The present application does not make specific limitations.

[0100] In determining the similar candidate text, the Average-Linkage clustering algorithm not only considers the tightness of the cluster, but also avoids excessive diffusion caused by a single outlier (as may occur in Single-Linkage) or too strict clustering (as may occur in Complete-Linkage); using record linking technology, the system can automatically identify the fuzzy matching keyword when processing interactive statements and match it with the most relevant field or value in the database, even if there are synonyms, spelling errors, etc. High matching accuracy can be achieved, thereby improving the understanding and processing ability of the query.

[0101] Subsequently, a second segmentation result of the interactive sentence is determined according to the similar candidate text, the corresponding pre-labeled part-of-speech, and the strictly matched keyword. The segmentation result includes the query candidate text and the part-of-speech. The part-of-speech at least includes a field and a value.

[0102] In other words, after obtaining the similar candidate text, the server can segment the fuzzy matched keyword and perform part-of-speech tagging by the above-mentioned segmentation method of the strictly matched keyword, to obtain a segmentation result. The segmentation result includes the database field or value corresponding to each word in the interactive sentence.

[0103] By the above-mentioned solution, the interactive sentence can be accurately segmented to obtain the database field or value corresponding to the interactive sentence, thereby facilitating the conversion of the SQL statement for query.

[0104] In S105, the server generates an interactive SQL query statement corresponding to the condition expression and the segmentation result based on a plurality of preset SQL statement rule matching templates, to determine the interactive response information corresponding to the interactive sentence based on the interactive SQL query statement.

[0105] In the embodiments of the present application, the interactive SQL query statement corresponding to the condition expression and the segmentation result is generated based on a plurality of preset SQL statement rule matching templates, which specifically includes:

[0106] Firstly, the server determines a plurality of preset SQL statement rule matching templates based on the predefined operation of the user. One preset SQL statement rule matching template corresponds to one type of database query. The preset SQL statement rule matching template includes a to-be-filled slot for inserting the content of the database query statement. Subsequently, the server matches the corresponding preset SQL statement rule matching template according to the condition expression and the segmentation result, and fills the corresponding to-be-filled slot to obtain the interactive SQL query statement. The condition expression includes the query condition corresponding to the interactive SQL query statement. The segmentation result includes the query candidate text corresponding to the query condition and used for constructing the interactive SQL query statement.

[0107] That is, the user can define a series of SQL statement rule matching templates in advance, each template corresponding to a specific type of query. The template can protect the basic result of the SQL statement, and the to-be-filled slot for inserting the specific value (i.e., the content of the database query statement). For example, for the interactive query "query all employee names", a template can be defined as follows: "SELECT<column name>FROM", wherein <column name> and are to-be-filled slots that need to be filled according to the actual query content corresponding to the interactive sentence.

[0108] The server obtains the condition expression and the word segmentation result, and obtains mapping of the keywords in the interactive sentence to specific elements of the database, including table name, column name, condition value, and the like. For example, the interactive sentence "find employees with an age greater than 30 years old" is mapped to the table name "employees", the column name "age", the condition "greater than", and the value "30 years old". The server uses the mapping result and an association list between the mapping result and each preset SQL interactive sentence rule matching template to select one or more preset SQL sentence rule matching templates, and fills the selected templates with the analysis result to generate an interactive SQL query statement. For example, the analysis result is the target field "name" and the table name "employees", and the filled SQL template is "SELECT name FROM employees". The preset SQL sentence rule matching template can be prepared by a user or a developer according to actual needs, and the application does not make a specific limitation.

[0109] According to the above scheme, the application can realize accurate word segmentation processing of natural language, and ensure that the interactive sentence can be successfully matched with the database field or value. Meanwhile, the condition expression in the interactive sentence and the corresponding database field or value are identified, which can guarantee the accuracy of the generated interactive SQL query statement, effectively convert the natural language into an executable SQL statement, avoid ambiguity in the conversion of the SQL statement, and realize generation of a complex logic SQL statement, thereby providing sufficient data preparation conditions for AIGC interactive analysis.

[0110] In addition, the application does not need to manually add synonyms when solving synonym identification, which improves the word segmentation and recognition efficiency. According to the technical scheme of the application, accurate multi-round dialogue can be realized in the process of generating AIGC answer content, and the latest answer elements can be monitored and updated in real time on the basis of the previous answer when continuous questioning, which can reduce the calculation process, improve the calculation efficiency, and increase the intelligence of the answer system.

[0111] In the embodiment of the application, the server combines the interactive SQL query statement with the ability of the AIGC to answer questions, integrates the query content into the AIGC answer, and provides interactive response information to the user terminal. The user terminal can be a mobile phone, computer, or other device of the user who issues the interactive sentence, and the application does not make a specific limitation.

[0112] When the application is applied to the AIGC dialogue BI system, the answer process for the natural language interactive sentence is as shown in FIG. 2, which includes text normalization, field / value identification, condition identification (i.e., condition expression identification), generation of a complete SQL and query (i.e., generation of an interactive SQL query statement), and AIGC answer in natural language.

[0113] FIG. 3 is a structural schematic diagram of a data preparation system for AIGC interaction analysis provided by an embodiment of the present application. As shown in FIG. 3, the data preparation system for AIGC interaction analysis 300 comprises:

[0114] The label sequence determination module 301 is configured to determine a label sequence corresponding to the first segmented interaction sentence by using a pre-trained text label recognition model. The label sequence comprises preset labels corresponding to each first text segment in the order of the interaction sentence. The condition expression determination module 302 is configured to determine a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence based on a preset expression construction rule. The preset expression construction rule is used to combine the plurality of preset labels in the label sequence to obtain the label combination sequence. Each label combination sequence has a unique corresponding condition expression. The keyword determination module 303 is configured to determine a strictly matched keyword and a fuzzy matched keyword corresponding to the second segmented interaction sentence by using a preconfigured database candidate text dictionary and a preset matching rule. The preset matching rule is used to match the candidate text in the database candidate text dictionary that is consistent with the second text segment. The sentence generation module 305 is configured to generate an interaction SQL query sentence corresponding to the condition expression and the segmentation result based on a plurality of preset SQL sentence rule matching templates, so as to determine an interaction response information corresponding to the interaction sentence based on the interaction SQL query sentence.

[0115] In the embodiment of the present application, before determining the label sequence corresponding to the first segmented interaction sentence by using the pre-trained text label recognition model, the data preparation system for AIGC interaction analysis 300 can also:

[0116] obtain a preconfigured text normalization mapping table. After receiving the interaction sentence from the user terminal, the text normalization mapping table is used to determine the text to be normalized in the interaction sentence, and the text format of the text to be normalized is mapped to a normalized format to obtain the interaction sentence in the normalized format. The normalized format at least includes a number normalization format, a time normalization format, a punctuation symbol normalization format, and a reserved word normalization format.

[0117] In the embodiment of the present application, before determining the label sequence corresponding to the first segmented interaction sentence by using the pre-trained text label recognition model, the data preparation system for AIGC interaction analysis 300 can also:

[0118] A pre-constructed sentence training sample set is obtained. The sentence training sample set includes at least a plurality of interactive sentence samples and corresponding SQL query statements. Each interactive sentence sample is pre-labeled with a preset label used to form a conditional expression. Each interactive sentence sample in the sentence training sample set is input into the text label recognition model to be trained, so that the text label recognition model performs first tokenization processing on each interactive sentence sample through the BERT model, and outputs a sample predicted label sequence corresponding to the first tokenization processed interactive sentence sample through the output layer. The difference degree value of each sample predicted label sequence and the corresponding true label sequence is determined through a cross-entropy loss function, until the difference degree value is less than a preset threshold, and a trained text label recognition model is obtained.

[0119] In the embodiment of the application, the keyword determination module 303 is specifically configured to:

[0120] The interactive sentence is subjected to second tokenization processing through the HanLP tokenizer, and the second text tokenization consistent with the candidate text is recognized through the database candidate text dictionary. The second text tokenization consistent with the candidate text is subjected to part-of-speech tagging according to the pre-labeled part-of-speech corresponding to the candidate text, so as to obtain the strictly matched keyword. The second text tokenization inconsistent with any candidate text is taken as the fuzzy matched keyword.

[0121] In the embodiment of the application, the tokenization result determination module 304 is specifically configured to:

[0122] The fuzzy matched keyword is converted into a keyword vector through the pre-trained word vector model. The text similarity between the keyword vector and the candidate text vector corresponding to each candidate text is calculated, and each text similarity is taken as an input value and input into a preset hierarchical clustering algorithm to determine a similar candidate text set corresponding to the fuzzy matched keyword. Each candidate text pair generated by the similar candidate text set and the fuzzy matched keyword is filtered through a preset filtering rule to obtain a similar candidate text corresponding to the fuzzy matched keyword. The tokenization result after the second tokenization processing on the interactive sentence is determined according to the similar candidate text, the pre-labeled part-of-speech corresponding to the similar candidate text, and the strictly matched keyword. The tokenization result includes the query candidate text and the part-of-speech thereof. The part-of-speech at least includes the field and the value.

[0123] In the embodiment of the application, the preset hierarchical clustering algorithm is an Average-Linkage clustering algorithm.

[0124] In the embodiment of the application, the sentence generation module 305 is specifically configured to:

[0125] Based on a predefined operation of a user, a plurality of preset SQL statement rule matching templates are determined. One preset SQL statement rule matching template corresponds to one database query type. The preset SQL statement rule matching template includes a to-be-filled slot for constituting a database query statement. According to a condition expression and a segmentation result, a corresponding preset SQL statement rule matching template is matched, and the corresponding to-be-filled slot is filled, to obtain an interactive SQL query statement. The condition expression includes a query condition corresponding to the interactive SQL query statement. The segmentation result includes a query candidate text corresponding to the query condition and used for constituting the interactive SQL query statement.

[0126] FIG. 4 is a structural schematic diagram of a data preparation device for AIGC interactive analysis provided by an embodiment of the present application. As shown in FIG. 4, the device includes:

[0127] at least one processor; and a memory communicatively connected with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0128] A label sequence corresponding to the first segmented interactive sentence is determined through a pre-trained text label recognition model. The label sequence includes preset labels corresponding to each first text segmentation and arranged in the order of the interactive sentence. Based on a preset expression construction rule, a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence are determined. The preset expression construction rule is used to combine a plurality of preset labels in the label sequence and obtain the label combination sequence. Each label combination sequence has a unique corresponding condition expression. Through a preconfigured database candidate text dictionary and a preset matching rule, strictly matched keywords and fuzzy matched keywords corresponding to the second segmented interactive sentence are determined. The preset matching rule is used to match candidate texts in the database candidate text dictionary that are consistent with the second text segmentation. Based on the strictly matched keywords and similar candidate texts corresponding to the fuzzy matched keywords, a segmentation result corresponding to the second segmentation processing is determined. Based on a plurality of preset SQL statement rule matching templates, an interactive SQL query statement corresponding to the condition expression and the segmentation result is generated, to determine interactive response information corresponding to the interactive sentence based on the interactive SQL query statement.

[0129] The embodiments of the present application also provide a non-volatile computer storage medium storing computer executable instructions, which are configured to:

[0130] The text label recognition model trained in advance is used to determine a label sequence corresponding to the first segmented interactive sentence. The label sequence includes preset labels corresponding to each first text segment in the order of the interactive sentence. Based on a preset expression construction rule, a label combination sequence corresponding to the label sequence and a conditional expression corresponding to the label combination sequence are determined. The preset expression construction rule is used to combine multiple preset labels in the label sequence and obtain the label combination sequence. Each label combination sequence has a unique corresponding conditional expression. Through a preconfigured database candidate text dictionary and a preset matching rule, strictly matched keywords and fuzzy matched keywords corresponding to the second segmented interactive sentence are determined. The preset matching rule is used to match candidate texts in the database candidate text dictionary that are consistent with the second text segment. Based on the strictly matched keywords and the similar candidate texts corresponding to the fuzzy matched keywords, a segmentation result corresponding to the second segmented interactive sentence is determined. Based on a plurality of preset SQL statement rule matching templates, an interactive SQL query statement corresponding to the conditional expression and the segmentation result is generated, so as to determine interactive response information corresponding to the interactive sentence based on the interactive SQL query statement.

[0131] Each embodiment in the present application is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments. In particular, for system, device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0132] The system, device and medium provided by the embodiments of the present application are one-to-one corresponding to the method, and therefore, the system, device and medium also have similar beneficial technical effects to the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system, device and medium will not be described here.

[0133] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the phrase "including a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0134] The above merely provides an example of the present application, but is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A data preparation method for AIGC interaction analysis, wherein, The method comprises: determining a label sequence corresponding to the first segmented interactive sentence through a pre-trained text label recognition model; wherein the label sequence comprises a preset label corresponding to each first text segmentation and arranged in the order of the interactive sentence; determining a label combination sequence corresponding to the label sequence and a condition expression corresponding to the label combination sequence based on a preset expression construction rule; the preset expression construction rule is used to combine multiple preset labels in the label sequence to obtain the label combination sequence; each label combination sequence has a unique corresponding condition expression; determining a strictly matched keyword and a fuzzy matched keyword corresponding to the second segmented interactive sentence through a preconfigured database candidate text dictionary and a preset matching rule; wherein the preset matching rule is used to match a candidate text consistent with the second text segmentation in the database candidate text dictionary; determining a segmentation result corresponding to the second segmentation processing based on the strictly matched keyword and a similar candidate text corresponding to the fuzzy matched keyword; generating an interactive SQL query statement corresponding to the condition expression and the segmentation result based on a plurality of preset SQL statement rule matching templates, and determining an interactive response information corresponding to the interactive sentence based on the interactive SQL query statement; wherein, before determining the label sequence corresponding to the first segmented interactive sentence through the pre-trained text label recognition model, the method further comprises: obtaining a pre-constructed sentence training sample set; wherein the sentence training sample set comprises at least a plurality of interactive sentence samples and their corresponding SQL query statements; each interactive sentence sample is pre-labeled with a preset label used to form the condition expression; inputting each interactive sentence sample in the sentence training sample set into a text label recognition model to be trained, so that the text label recognition model performs the first segmentation processing on each interactive sentence sample through its BERT model, and outputs a sample predicted label sequence corresponding to the first segmented interactive sentence sample through the output layer; determining the difference degree value of each sample predicted label sequence and the corresponding real label sequence through a cross-entropy loss function until the difference degree value is less than a preset threshold, thereby obtaining the trained text label recognition model; wherein, the preset expression construction rule comprises the association between the label combination sequence obtained by combining different preset labels and different condition expressions.

2. The data preparation method for AIGC interaction analysis according to claim 1, wherein, Before determining the label sequence corresponding to the first segmented interactive sentence through the pre-trained text label recognition model, the method further comprises: obtaining a pre-constructed text normalization mapping table; After receiving the interactive sentence from the user terminal, the text to be standardized in the interactive sentence is determined through the text standardization mapping table, and the text format of the text to be standardized is mapped to a standard format to obtain the interactive sentence in the standard format; the standard format at least includes a number standard format, a time standard format, a punctuation symbol standard format, and a reserved word standard format.

3. The data preparation method for AIGC interaction analysis according to claim 1 or 2, wherein, Through a pre-set database candidate text dictionary and a pre-set matching rule, strict matching keywords and fuzzy matching keywords corresponding to the interactive sentence subjected to the second word segmentation processing are determined, specifically including: The second word segmentation processing is performed on the interactive sentence through a HanLP word segmenter, and the second text segmentation consistent with the candidate text is identified through the database candidate text dictionary; According to the pre-labeled part-of-speech corresponding to the candidate text, the second text segmentation consistent with the candidate text is subjected to part-of-speech tagging to obtain the strict matching keywords; The second text segmentation inconsistent with any of the candidate texts is taken as the fuzzy matching keywords.

4. The data preparation method for AIGC interaction analysis according to claim 1, wherein, Based on the strict matching keywords and the similar candidate texts corresponding to the fuzzy matching keywords, a word segmentation result corresponding to the second word segmentation processing is determined, specifically including: The fuzzy matching keywords are converted into keyword vectors through a pre-trained word vector model; Text similarities between the keyword vectors and candidate text vectors corresponding to each of the candidate texts are calculated, and each of the text similarities is taken as an input value and input into a pre-set hierarchical clustering algorithm to determine a similar candidate text set corresponding to the fuzzy matching keywords; Each candidate text pair generated by the similar candidate text set and the fuzzy matching keywords is filtered through a pre-set filtering rule to obtain the similar candidate text corresponding to the fuzzy matching keywords; According to the similar candidate text, the pre-labeled part-of-speech corresponding thereto, and the strict matching keywords, the word segmentation result after the second word segmentation processing on the interactive sentence is determined; the word segmentation result includes a query candidate text and a part-of-speech thereof; the part-of-speech at least includes a field and a value.

5. The data preparation method for AIGC interaction analysis according to claim 4, wherein, The pre-set hierarchical clustering algorithm is an Average-Linkage clustering algorithm.

6. The data preparation method for AIGC interaction analysis according to claim 1, wherein, Based on a plurality of pre-set SQL statement rule matching templates, an interactive SQL query statement corresponding to the condition expression and the word segmentation result is generated, specifically including: Based on a user's pre-defined operation, a plurality of the pre-set SQL statement rule matching templates are determined; one of the pre-set SQL statement rule matching templates corresponds to one database query type; the pre-set The SQL statement rule matching template includes a to-be-filled slot for constituting a database query statement; According to the condition expression and the word segmentation result, a corresponding pre-set SQL statement rule matching template is matched, and a corresponding to-be-filled slot is filled to obtain the interactive SQL query statement; wherein, the condition expression includes a query condition corresponding to the interactive SQL query statement; the word segmentation result includes a query candidate text corresponding to the query condition and used for constituting the interactive SQL query statement.

7. A data preparation system for AIGC interaction analysis, wherein, The system comprises: A label sequence determination module is configured to determine a label sequence corresponding to an interactive sentence after first word segmentation by using a pre-trained text label recognition model; wherein the label sequence comprises preset labels corresponding to each first text word segmentation and arranged in the order of the interactive sentence; A conditional expression determination module is configured to determine a label combination sequence corresponding to the label sequence and a conditional expression corresponding to the label combination sequence based on a preset expression construction rule; the preset expression construction rule is used to combine multiple preset labels in the label sequence to obtain the label combination sequence; each label combination sequence has a unique corresponding conditional expression; A keyword determination module is configured to determine strictly matched keywords and fuzzy matched keywords corresponding to the interactive sentence after second word segmentation by using a pre-configured database candidate text dictionary and a preset matching rule; wherein the preset matching rule is used to match candidate texts consistent with second text word segmentation in the database candidate text dictionary; A word segmentation result determination module is configured to determine a word segmentation result corresponding to the second word segmentation based on the strictly matched keywords and similar candidate texts corresponding to the fuzzy matched keywords; A sentence generation module is configured to generate an interactive SQL query sentence corresponding to the conditional expression and the word segmentation result based on a plurality of preset SQL sentence rule matching templates, and determine an interactive response information corresponding to the interactive sentence based on the interactive SQL query sentence; Before determining the label sequence corresponding to the interactive sentence after first word segmentation by using the pre-trained text label recognition model, the system can also: Obtain a pre-constructed sentence training sample set; wherein the sentence training sample set comprises at least a plurality of interactive sentence samples and their corresponding SQL query sentences; each interactive sentence sample is pre-labeled with the preset labels used to form the conditional expression; Input each interactive sentence sample in the sentence training sample set into a text label recognition model to be trained, so that the text label recognition model performs the first word segmentation on each interactive sentence sample through its BERT model, and outputs a sample predicted label sequence corresponding to the interactive sentence sample after the first word segmentation through the output layer; Determine the difference degree value of each sample predicted label sequence and the corresponding real label sequence by using a cross-entropy loss function, until the difference degree value is less than a preset threshold, and obtain the trained text label recognition model; The preset expression construction rule comprises the association between the label combination sequence obtained by combining different preset labels and different conditional expressions. The device comprises:

8. A data preparation device for AIGC interaction analysis, wherein, At least one processor; and A memory connected in communication with the at least one processor; wherein ​ The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a data preparation method for AIGC interaction analysis according to any one of claims 1-6.

9. A non-transitory computer storage medium storing computer-executable instructions, wherein, The computer executable instructions are capable of performing a data preparation method for AIGC interaction analysis according to any one of claims 1-6.

Images