Intelligent question answering methods, devices, and related equipment based on the BERT model

Abstract

This application relates to the field of artificial intelligence technology, and provides an intelligent question answering method, apparatus, and related equipment based on the BERT model. The method includes: responding to a received intelligent question answering request, obtaining a target question identified by aggregation operations and an association table of the target question; obtaining a first word embedding vector of the target question; obtaining a second word embedding vector of the association table; concatenating the first word embedding vector and the second word embedding vector to obtain a third word embedding vector; inputting the third word embedding vector into a pre-trained BERT model, and outputting the probability value of each aggregation operation and the probability value of each cell in the association table; and obtaining the answer to the target question based on the probability values ​​of multiple aggregation operations and the probability values ​​of multiple cells. This application improves the feedback efficiency of intelligent question answering by inputting the third word embedding vector into a pre-trained BERT model to obtain the answer to the target question.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an intelligent question-answering method, device, and related equipment based on the BERT model. Background Technology

[0002] Automatic question answering systems are a very important research hotspot in the field of natural language processing and have significant application value. Traditional techniques translate natural language questions into a database query statement similar to SQL, and then execute the SQL query statement to provide an answer.

[0003] However, this method typically requires extensive data engineering to generate database query statements and is difficult to extend to arbitrary tables or questions, resulting in low efficiency in providing question answers. Summary of the Invention

[0004] In view of the above, it is necessary to propose an intelligent question answering method, device and related equipment based on the BERT model. By inputting the third word embedding vector into the pre-trained BERT model, the answer to the target question is obtained, which improves the feedback efficiency of intelligent question answering.

[0005] The first aspect of this application provides an intelligent question answering method based on the BERT model, the method comprising:

[0006] In response to a received intelligent question-answering request, obtain the target question that is identified by an aggregation operation and the associated table of the target question;

[0007] Obtain the first word embedding vector of the target question;

[0008] Obtain the second word embedding vector of the association table;

[0009] The first word embedding vector and the second word embedding vector are concatenated to obtain the third word embedding vector;

[0010] The third word embedding vector is input into the pre-trained BERT model, and the probability value of each aggregation operation and the probability value of each cell in the association table are output.

[0011] The answer to the target question is obtained based on the probability values ​​of multiple aggregation operations and multiple cells.

[0012] Optionally, obtaining the first word embedding vector of the target question includes:

[0013] The target problem is segmented into words to obtain the word code for each word;

[0014] Obtain the position code of each word in the target question;

[0015] Obtain the sentence label code and rank sort code for each word;

[0016] Obtain the column code and row code for each word;

[0017] The word encoding, the position encoding, the sentence number encoding, the column encoding, the row encoding, and the rank-sorted encoding are superimposed to form the first word embedding vector.

[0018] Optionally, obtaining the second word embedding vector of the association table includes:

[0019] The associated table is divided into header text and table text;

[0020] The header text and table text are segmented into words to obtain the word code, position code, sentence number code, column code, row code, and rank sort code for each word.

[0021] The first sub-embedding vector is formed by superimposing the word code, position code, sentence number code, column code, row code, and rank sort code corresponding to the header text;

[0022] The word code, position code, sentence number code, column code, row code, and rank-sorted code corresponding to the table text are superimposed to form a second sub-embedding vector;

[0023] The first sub-embedding vector and the second sub-embedding vector are concatenated to obtain the second word embedding vector.

[0024] Optionally, before inputting the third word embedding vector into the pre-trained BERT model and outputting the probability value of each aggregation operation and the probability value of each cell in the association table, the method further includes:

[0025] Create a pre-trained BERT model, wherein the pre-trained BERT model includes a word embedding layer, a position embedding layer, a sentence label embedding layer, a column index embedding layer, a row index embedding layer, and a rank-ordered embedding layer, and the first label output position of the BERT model is connected to a softmax output layer;

[0026] Retrieve a set of historical questions marked with aggregation operation identifiers and the associated table for each historical question from a preset data source;

[0027] The set of historical questions and the association table of the set of historical questions are used as the training set;

[0028] The pre-trained BERT model is trained based on the training set to obtain the BERT model.

[0029] Optionally, using the historical problem set and its association table as a training set includes:

[0030] Retrieve each historical question and its associated table;

[0031] A preset number of word fragments are randomly selected from each historical question, and tokenized text is generated based on the preset number of word fragments;

[0032] Extract the first character from the association table corresponding to the historical issues;

[0033] Following the extraction order, the first word of each table is added to the end of the tokenized text;

[0034] If the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, the addition stops, and the word fragment sequence corresponding to the historical question is obtained.

[0035] If the length of the combined text of the tokenized text and the added table units does not meet the preset word fragment sequence length threshold, the second character of each table is extracted from the association table, and the above addition process is repeated until the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, thus obtaining the word fragment sequence of the corresponding historical question.

[0036] The set of word fragment sequences from the historical question set is used as the training set.

[0037] Optionally, obtaining the answer to the target question based on the probability values ​​of multiple aggregation operations and the probability values ​​of multiple cells includes:

[0038] Select the aggregation operation with the highest probability from the probability values ​​of the plurality of aggregation operations;

[0039] The probability of each cell is compared with a preset probability threshold;

[0040] Filter out the text in cells whose probability is greater than or equal to the preset probability threshold;

[0041] Perform the corresponding aggregation operation on the text to obtain the answer to the target question.

[0042] Optionally, after obtaining the target problem identified by the aggregation operation and the association table of the target problem, the method further includes:

[0043] In response to the intelligent question-answering request, obtain message information;

[0044] Obtain the problem text from the message information;

[0045] Insert a first marker at the beginning of the question text and a second marker at the end of the question text;

[0046] The marked question text is identified as the target question.

[0047] A second aspect of this application provides an intelligent question-answering device based on the BERT model, the device comprising:

[0048] The first acquisition module is used to, in response to a received intelligent question-answering request, acquire the target question identified by the aggregation operation and the association table of the target question;

[0049] The second acquisition module is used to acquire the first word embedding vector of the target question;

[0050] The third acquisition module is used to acquire the second word embedding vector of the association table;

[0051] The determining module is used to concatenate the first word embedding vector and the second word embedding vector to obtain the third word embedding vector;

[0052] The input module is used to input the third word embedding vector into the pre-trained BERT model and output the probability value of each aggregation operation and the probability value of each cell of the association table.

[0053] The fourth acquisition module is used to obtain the answer to the target question based on the probability values ​​of multiple aggregation operations and the probability values ​​of multiple cells.

[0054] A third aspect of this application provides an electronic device including a processor and a memory, wherein the processor is configured to implement the BERT-based intelligent question-answering method by executing a computer program stored in the memory.

[0055] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the BERT-based intelligent question-answering method.

[0056] In summary, the intelligent question-answering method, device, and related equipment based on the BERT model described in this application can promote the construction of smart cities and be applied to fields such as smart buildings, smart security, smart communities, smart living, and the Internet of Things. Since the third word embedding vector is obtained by concatenating the first word embedding vector of the target question (identifying aggregation operations) and the second word embedding vector of the target question's association table, by inputting the third word embedding vector into a pre-trained BERT model, the BERT model introduces targeted architectural adjustments based on the characteristics of tabular data. Relying on a weakly supervised approach, it can output the probability value of each aggregation operation and the probability value of each cell in the association table, thus directly obtaining the final answer to the target question. Furthermore, the model learns the possible aggregation operations and cell selections itself, which is a more useful design pattern. This not only solves the problem of manual labor costs but also makes the model more flexible, avoids SQL rule design, and improves the feedback efficiency of intelligent question-answering answers. Attached Figure Description

[0057] Figure 1 This is a flowchart of the intelligent question answering method based on the BERT model provided in Embodiment 1 of this application.

[0058] Figure 2 This is a schematic diagram of the embedding layer of the BERT model provided in Embodiment 1 of this application.

[0059] Figure 3 This is a schematic diagram of the output result of the BERT model provided in Embodiment 1 of this application.

[0060] Figure 4 This is a structural diagram of the intelligent question-answering device based on the BERT model provided in Embodiment 2 of this application.

[0061] Figure 5 This is a schematic diagram of the structure of the electronic device provided in Embodiment 3 of this application. Detailed Implementation

[0062] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0064] In the field of intelligent problem-solving technology, natural language problems are generally translated into a database query statement similar to SQL, and then the SQL query statement is executed to provide the answer. If the natural language problem is: "How many wrestlers have won the championship only once?", the problem will be transformed into a query like "select count(*) where column("number of championships") ==1;", and then the statement will be executed to produce the answer.

[0065] These methods typically require extensive data engineering to generate SQL queries that are syntactically and semantically valid, and are difficult to scale to arbitrary tables or questions. They not only require a lot of human intervention but also lack flexibility and are difficult to scale.

[0066] This application proposes an intelligent question-answering method based on the BERT model. This method is an end-to-end training approach that extends the BERT architecture by encoding questions along with tabular data structures, thereby generating a model that directly points to the answer. This method can be applied to models from tables across a wide range of domains. After pre-training on millions of Wikipedia tables, the trained model can quickly provide answers to questions, demonstrating high flexibility.

[0067] Example 1

[0068] Figure 1 This is a flowchart of the intelligent question answering method based on the BERT model provided in Embodiment 1 of this application.

[0069] In this embodiment, the intelligent question answering method based on the BERT model can be applied to electronic devices. For electronic devices that need to perform intelligent question answering based on the BERT model, the intelligent question answering function based on the BERT model provided by the method of this application can be directly integrated into the electronic device, or it can be run in the electronic device in the form of a software development kit (SDK).

[0070] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0071] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0072] like Figure 1 As shown, the intelligent question answering method based on the BERT model specifically includes the following steps. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0073] 101. In response to the received intelligent question answering request, obtain the target question that is identified by the aggregation operation and the association table of the target question.

[0074] In this embodiment, intelligent question answering refers to automatically determining the answer based on the question. When the electronic device receives an intelligent question answering request, it parses the intelligent question request and further obtains the target question and the association table of the target question.

[0075] In an optional embodiment, after obtaining the target problem identified by the aggregation operation and the association table of the target problem, the method further includes:

[0076] In response to the intelligent question-answering request, obtain message information;

[0077] Obtain the problem text from the message information;

[0078] Insert a first marker at the beginning of the question text and a second marker at the end of the question text;

[0079] The marked question text is identified as the target question.

[0080] In this embodiment, the first marker can be the [CLS] marker, and the second marker can be the [SEP] marker.

[0081] In this embodiment, when the target question and its associated table are obtained, the traditional BERT model based on Transformers is extended by encoding the target question and the table text of the associated table together to generate a BERT model that directly points to the answer to the target question.

[0082] In this embodiment, since the BERT model expects the input data to have a specific format, special markers are used to mark the beginning ([CLS]) and the separator / end ([SEP]) of the question text.

[0083] 102, Obtain the first word embedding vector of the target question.

[0084] In an optional embodiment, obtaining the first word embedding vector of the target question includes:

[0085] The target problem is segmented into words to obtain the word code for each word;

[0086] Obtain the position code of each word in the target question;

[0087] Obtain the sentence label code and rank sort code for each word;

[0088] Obtain the column code and row code for each word;

[0089] The word encoding, the position encoding, the sentence number encoding, the column encoding, the row encoding, and the rank-sorted encoding are superimposed to form the first word embedding vector.

[0090] In this embodiment, please refer to Table 1 and... Figure 2 As shown, Figure 2 In this context, Token Embeddings represents word encoding; Position Embeddings represents position encoding; Segment Embeddings represents sentence label encoding; Column Embeddings represents column encoding; Row Embeddings represents row encoding; and Rank Embeddings represents rank-order encoding.

[0091] In this embodiment, after obtaining the target question, the input parameters of the question part of the BERT model are obtained, namely the first word embedding vector, for example, E. [CLS] =[CLS]+Pos0+seg0+col0+row0+rank0.

[0092]

[0093] Table 1

[0094] In this embodiment, the position encoding is used to characterize the position of each word in the target question.

[0095] In this embodiment, rank is a parameter used to parse floating-point numbers or date data so that it can be sorted. Different embedding representations are assigned to them according to their numerical size, with 1 representing the smallest item and the larger the number, the larger the value. Rank0 represents an incomparable item, cell 3749 represents rank2, cell 3103 represents rank1, and all other items are rank0.

[0096] In this embodiment, since there is no table in the target problem, the sentence number encoding of the target problem is set to seg0; the column encoding of the target problem is set to col0; and the row encoding of the target problem is set to row0.

[0097] 103. Obtain the second word embedding vector of the association table.

[0098] In an optional embodiment, obtaining the second word embedding vector of the association table includes:

[0099] The associated table is divided into header text and table text;

[0100] The header text and table text are segmented into words to obtain the word code, position code, sentence number code, column code, row code, and rank sort code for each word.

[0101] The first sub-embedding vector is formed by superimposing the word code, position code, sentence number code, column code, row code, and rank sort code corresponding to the header text;

[0102] The word code, position code, sentence number code, column code, row code, and rank-sorted code corresponding to the table text are superimposed to form a second sub-embedding vector;

[0103] The first sub-embedding vector and the second sub-embedding vector are concatenated to obtain the second word embedding vector.

[0104] In this embodiment, since the association table contains header text and table text, the association table is divided into a header part and a table part.

[0105] In this embodiment, the sentence identifier code of both the header text and the table text is set to seg1.

[0106] In this embodiment, the column code and the row code are used to indicate the position of the cell in the associated table.

[0107] In this embodiment, the second word embedding vector is obtained by horizontally concatenating the first sub-embedding vector and the second sub-embedding vector.

[0108] 104. The first word embedding vector and the second word embedding vector are concatenated to obtain the third word embedding vector.

[0109] In this embodiment, since there are word embedding vectors for both the target question part and the association table, the third word embedding vector is obtained by horizontally concatenating the first word embedding vector and the second word embedding vector.

[0110] 105. Input the third word embedding vector into the pre-trained BERT model and output the probability value of each aggregation operation and the probability value of each cell in the association table.

[0111] In this embodiment, after obtaining the third word embedding vector, the created BERT model can be pre-trained. Specifically, the training process of the BERT model includes:

[0112] Create a pre-trained BERT model, wherein the pre-trained BERT model includes a word embedding layer, a position embedding layer, a sentence label embedding layer, a column index embedding layer, a row index embedding layer, and a rank-ordered embedding layer, and the first label output position of the BERT model is connected to a softmax output layer;

[0113] Retrieve a set of historical questions marked with aggregation operation identifiers and the associated table for each historical question from a preset data source;

[0114] The set of historical questions and the association table of the set of historical questions are used as the training set;

[0115] The pre-trained BERT model is trained based on the training set to obtain the BERT model.

[0116] In this embodiment, the preset data source can be a database such as XX Encyclopedia that contains historical questions and related tables of historical questions.

[0117] In this embodiment, the aggregation operation identifier is used to uniquely identify the aggregation operation, which includes: summation (SUM), average (AVERAGE), counting (COUNT), and no aggregation operation (NONE).

[0118] In this embodiment, see Figure 2 As shown, in order to encode the target question and the table text of the corresponding association table together, the traditional BERT model is extended for the table results by learning and embedding column indexes, row indexes and a special rank index, to generate a BERT model that directly points to the answer to the target question.

[0119] In this embodiment, see Figure 3 As shown, [E [CLS] E1…E N E [SEP] E c …E c ] represents the third word embedding vector, Q1 to Q N This represents the vector corresponding to [CLS], from C1 to C... M This indicates the output probability of different cells.

[0120] In this embodiment, the pre-trained BERT model is similar in architecture to the original BERT model. The first token ([CLS]) position of the last layer of the BERT model is used to predict the probability of the aggregation operation, and the predicted probability value of each aggregation operation is output. The output of the intermediate problem encoding part can be ignored, C1 to C M This indicates the output probability of different cells.

[0121] Specifically, the pre-trained BERT model has two outputs: the first result is the aggregation operation probability, indicating which aggregation operation to apply to generate the final answer; the second result is the output score for each cell in the association table, which represents the probability that the cell will become part of the answer.

[0122] Furthermore, using the historical problem set and its association table as a training set includes:

[0123] Retrieve each historical question and its associated table;

[0124] A preset number of word fragments are randomly selected from each historical question, and tokenized text is generated based on the preset number of word fragments;

[0125] Extract the first character from the association table corresponding to the historical issues;

[0126] Following the extraction order, the first word of each table is added to the end of the tokenized text;

[0127] If the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, the addition stops, and the word fragment sequence corresponding to the historical question is obtained.

[0128] If the length of the combined text of the tokenized text and the added table units does not meet the preset word fragment sequence length threshold, the second character of each table is extracted from the association table, and the above addition process is repeated until the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, thus obtaining the word fragment sequence of the corresponding historical question.

[0129] The set of word fragment sequences from the historical question set is used as the training set.

[0130] In this embodiment, in order to better enable the BERT model to learn and find the correspondence between the cells where the questions and answers are located, each historical question and its associated table are processed to obtain a set of word fragment sequences.

[0131] In this embodiment, in order to make pre-training effective, each historical question and its associated table are processed to limit the word fragment sequence to a preset word fragment sequence length threshold. The word fragment sequence length threshold can be set to 128 or 256, depending on the actual situation of the model.

[0132] For example, the preset threshold for the word fragment sequence length is set to 128, the length of the historical question is 50, and there are 100 association tables associated with the historical questions. First, 8-16 words are randomly selected from the historical questions as tokenized text. Then, from the 100 tables, the first word in each table is selected, and each word is added sequentially according to the extraction order to obtain the combined text of the table units. The length of the combined text of the tokenized text and the table units is calculated. If the length reaches 128, a word fragment sequence is obtained. If the length does not reach 128, the second word in each table is selected and added until the length reaches 128. One association table can generate multiple word fragment sequences.

[0133] In this embodiment, each word is added sequentially according to the extraction order to obtain the combined text of the table units. This solves the problem that the content of the table is too long to fit at once when directly pasted, and the problem that adding one column at a time will prevent the subsequent columns from being added. This ensures that each table in the problem association table can be covered, improves the accuracy of word fragment sequences, and thus improves the accuracy of the training set.

[0134] In this embodiment, by processing the historical questions and association tables into a set of word fragment sequences to train the BERT model, the BERT model can better learn to find the correspondence between questions and answers in the corresponding cells, which improves the robustness of the BERT model and thus improves the output efficiency and accuracy of the first and second results.

[0135] 106. Based on the probability values ​​of multiple aggregation operations and multiple cells, obtain the answer to the target question.

[0136] In this embodiment, see Figure 3 As shown, the first result contains the probability value for each aggregation operation, and the second result contains the probability value for each cell.

[0137] In an optional embodiment, obtaining the answer to the target question based on the probability values ​​of multiple aggregation operations and the probability values ​​of multiple cells includes:

[0138] Select the aggregation operation with the highest probability from the probability values ​​of the plurality of aggregation operations;

[0139] The probability of each cell is compared with a preset probability threshold;

[0140] Filter out the text in cells whose probability is greater than or equal to the preset probability threshold;

[0141] Perform the corresponding aggregation operation on the text to obtain the answer to the target question.

[0142] In this embodiment, a probability threshold can be preset for the cells of the association table. The probability threshold can be set to 0.5. The probability threshold can be set according to the target problem and the actual situation of the association table. This embodiment does not impose any restrictions on this.

[0143] For example, with a preset probability threshold of 0.5, the target question is "What is the average championship time of the top two wrestlers?", the correlation table is shown in Table 1, and the first and second output results are as follows: Figure 3 As shown, the aggregation operation AVERAGE with the highest probability P is selected from the first result. The text of cells with a probability greater than or equal to the preset probability threshold is filtered out as 3749 and 3103. The answer to the target question is determined to be: "AVERAGE(3749,3103)=3426".

[0144] In this embodiment, during the fine-tuning of the BERT model, how the BERT model learns to answer questions in a table is accomplished using weakly supervised training. In strongly supervised training, given a table and questions, selected cells and aggregation operations (such as summation or counting) must be provided, which is a time-consuming and laborious process. With weakly supervised training, the training data can only provide the correct answers (e.g., question 3426 in the example table above). In this case, the model attempts to find an aggregation operation and a cell that produces a close approximation of the correct answer. This is done by calculating the expectation of all possible aggregation decisions and comparing it to the true result. Weakly supervised scenarios are beneficial because they allow non-experts to provide the data needed to train the model and require less time than strongly supervised training.

[0145] In this embodiment, by introducing targeted architectural adjustments based on the characteristics of tabular data into the traditional BERT model, but still using the original BERT pre-training or fine-tuning methods, the model can be trained on a large amount of tabular text and further fine-tuned on targeted downstream task tables to achieve higher performance, improve the robustness of the BERT model, and thus improve the output efficiency and accuracy of the first and second results.

[0146] In summary, the intelligent question answering method based on the BERT model described in this embodiment, by inputting the third word embedding vector into a pre-trained BERT model, outputs the probability value of each aggregation operation and the probability value of each cell in the associated table. The BERT model incorporates targeted architectural adjustments based on the characteristics of tabular data, relying on a weakly supervised approach to directly obtain the final answer to the target question. The model learns the aggregation operations and cell selections that may be involved, which is a more useful design pattern. It not only solves the problem of manual costs but also makes the model more flexible, avoids the design of SQL rules, and improves the feedback efficiency of intelligent question answering.

[0147] Example 2

[0148] Figure 4 This is a structural diagram of the intelligent question-answering device based on the BERT model provided in Embodiment 2 of this application.

[0149] In some embodiments, the BERT-based intelligent question-answering device 20 may include multiple functional modules composed of program code segments. The program code of each program segment in the BERT-based intelligent question-answering device 20 may be stored in the memory of the electronic device and executed by the at least one processor to perform (see details). Figures 1 to 3 (Description) Intelligent question answering functionality based on the BERT model.

[0150] In this embodiment, the BERT-based intelligent question-answering device 20 can be divided into multiple functional modules according to its functions. These functional modules may include: a first acquisition module 201, a second acquisition module 202, a third acquisition module 203, a determination module 204, an input module 205, and a fourth acquisition module 206. As used in this application, a module refers to a series of computer-readable instruction segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0151] The first acquisition module 201 is used to acquire, in response to the received intelligent question answering request, a target question that is identified by an aggregation operation and a related table of the target question.

[0152] The second acquisition module 202 is used to acquire the first word embedding vector of the target question.

[0153] The third acquisition module 203 is used to acquire the second word embedding vector of the association table.

[0154] The determining module 204 is used to concatenate the first word embedding vector and the second word embedding vector to obtain the third word embedding vector.

[0155] The input module 205 is used to input the third word embedding vector into the pre-trained BERT model and output the probability value of each aggregation operation and the probability value of each cell of the association table.

[0156] The fourth acquisition module 206 is used to obtain the answer to the target question based on the probability values ​​of multiple aggregation operations and the probability values ​​of multiple cells.

[0157] In an optional embodiment, after obtaining the target question identified by the aggregation operation and the association table of the target question, in response to the intelligent question answering request, message information is obtained; question text is obtained from the message information; a first marker is inserted at the beginning of the question text and a second marker is inserted at the end of the question text; and the marked question text is identified as the target question.

[0158] In an optional embodiment, the second acquisition module 202 is used to:

[0159] The target problem is segmented to obtain the word code of each word; the position code of each word in the target problem is obtained; the sentence number code and rank sort code of each word are obtained; the column code and row code of each word are obtained; the word code, the position code, the sentence number code, the column code, the row code and the rank sort code are superimposed to form the first word embedding vector.

[0160] In an optional embodiment, the third acquisition module 203 is used for:

[0161] The association table is divided into header text and table text. The header text and table text are then segmented into words to obtain the word code, position code, sentence number code, column code, row code, and rank-sort code for each word. The word code, position code, sentence number code, column code, row code, and rank-sort code corresponding to the header text are superimposed to form a first sub-embedding vector. The word code, position code, sentence number code, column code, row code, and rank-sort code corresponding to the table text are superimposed to form a second sub-embedding vector. The first sub-embedding vector and the second sub-embedding vector are concatenated to obtain a second word embedding vector.

[0162] In an optional embodiment, before inputting the third word embedding vector into a pre-trained BERT model and outputting the probability value of each aggregation operation and the probability value of each cell in the association table, a pre-trained BERT model is created. The pre-trained BERT model includes a word embedding layer, a position embedding layer, a sentence label embedding layer, a column index embedding layer, a row index embedding layer, and a rank-sorted embedding layer. The first label output position of the BERT model is connected to a softmax output layer. A set of historical questions with aggregation operation identifiers and an association table for each historical question are obtained from a preset data source. The set of historical questions and the association table of the historical questions are used as a training set. The pre-trained BERT model is trained based on the training set to obtain the BERT model.

[0163] In this embodiment, the preset data source can be a database such as XX Encyclopedia that contains historical questions and related tables of historical questions.

[0164] In this embodiment, the aggregation operation identifier is used to uniquely identify the aggregation operation, which includes: summation (SUM), average (AVERAGE), counting (COUNT), and no aggregation operation (NONE).

[0165] In this embodiment, see Figure 2 As shown, in order to encode the target question and the table text of the corresponding association table together, the traditional BERT model is extended for the table results by learning and embedding column indexes, row indexes and a special rank index, to generate a BERT model that directly points to the answer to the target question.

[0166] In this embodiment, see Figure 3 As shown, [E [CLS] E1…E N E [SEP] E c …E c ] represents the third word embedding vector, Q1 to Q N This represents the vector corresponding to [CLS], from C1 to C... M This indicates the output probability of different cells.

[0167] In this embodiment, the pre-trained BERT model is similar in architecture to the original BERT model. The first token ([CLS]) position of the last layer of the BERT model is used to predict the probability of the aggregation operation, and the predicted probability value of each aggregation operation is output. The output of the intermediate problem encoding part can be ignored, C1 to C M This indicates the output probability of different cells.

[0168] Specifically, the pre-trained BERT model has two outputs: the first result is the aggregation operation probability, indicating which aggregation operation to apply to generate the final answer; the second result is the output score for each cell in the association table, which represents the probability that the cell will become part of the answer.

[0169] Further, using the historical question set and its association table as a training set includes: obtaining each historical question and its association table; randomly selecting a preset number of word fragments from each historical question, and generating tokenized text based on the preset number of word fragments; extracting the first character of each table from the association table of the corresponding historical question; adding the first character of each table to the end of the tokenized text in the extraction order; stopping the addition if the length of the combined text of the tokenized text and the added table units meets a preset word fragment sequence length threshold, thus obtaining a word fragment sequence for the corresponding historical question; if the length of the combined text of the tokenized text and the added table units does not meet the preset word fragment sequence length threshold, extracting the second character of each table from the association table, and repeating the above addition process until the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, thus obtaining a word fragment sequence for the corresponding historical question; and using the word fragment sequence set of the historical question set as a training set.

[0170] In this embodiment, in order to better enable the BERT model to learn and find the correspondence between the cells where the questions and answers are located, each historical question and its associated table are processed to obtain a set of word fragment sequences.

[0171] In this embodiment, in order to make pre-training effective, each historical question and its associated table are processed to limit the word fragment sequence to a preset word fragment sequence length threshold. The word fragment sequence length threshold can be set to 128 or 256, depending on the actual situation of the model.

[0172] For example, the preset threshold for the word fragment sequence length is set to 128, the length of the historical question is 50, and there are 100 association tables associated with the historical questions. First, 8-16 words are randomly selected from the historical questions as tokenized text. Then, from the 100 tables, the first word in each table is selected, and each word is added sequentially according to the extraction order to obtain the combined text of the table units. The length of the combined text of the tokenized text and the table units is calculated. If the length reaches 128, a word fragment sequence is obtained. If the length does not reach 128, the second word in each table is selected and added until the length reaches 128. One association table can generate multiple word fragment sequences.

[0173] In this embodiment, each word is added sequentially according to the extraction order to obtain the combined text of the table units. This solves the problem that the content of the table is too long to fit at once when directly pasted, and the problem that adding one column at a time will prevent the subsequent columns from being added. This ensures that each table in the problem association table can be covered, improves the accuracy of word fragment sequences, and thus improves the accuracy of the training set.

[0174] In this embodiment, by processing the historical questions and association tables into a set of word fragment sequences to train the BERT model, the BERT model can better learn to find the correspondence between questions and answers in the corresponding cells, which improves the robustness of the BERT model and thus improves the output efficiency and accuracy of the first and second results.

[0175] In an optional embodiment, the fourth acquisition module 206 is used to:

[0176] Select the aggregation operation with the highest probability from the probability values ​​of the multiple aggregation operations; compare the probability of each cell with a preset probability threshold; filter out the text of cells with a probability greater than or equal to the preset probability threshold; perform the corresponding aggregation operation on the text to obtain the answer to the target question.

[0177] For example, with a preset probability threshold of 0.5, the target question is "What is the average championship time of the top two wrestlers?", the correlation table is shown in Table 1, and the first and second output results are as follows: Figure 3 As shown, the aggregation operation AVERAGE with the highest probability P is selected from the first result. The text of cells with a probability greater than or equal to the preset probability threshold is filtered out as 3749 and 3103. The answer to the target question is determined to be: "AVERAGE(3749,3103)=3426".

[0178] In this embodiment, during the fine-tuning of the BERT model, how the BERT model learns to answer questions in a table is accomplished using weakly supervised training. In strongly supervised training, given a table and questions, selected cells and aggregation operations (such as summation or counting) must be provided, which is a time-consuming and laborious process. With weakly supervised training, the training data can only provide the correct answers (e.g., question 3426 in the example table above). In this case, the model attempts to find an aggregation operation and a cell that produces a close approximation of the correct answer. This is done by calculating the expectation of all possible aggregation decisions and comparing it to the true result. Weakly supervised scenarios are beneficial because they allow non-experts to provide the data needed to train the model and require less time than strongly supervised training.

[0179] In this embodiment, by introducing targeted architectural adjustments based on the characteristics of tabular data into the traditional BERT model, but still using the original BERT pre-training or fine-tuning methods, the model can be trained on a large amount of tabular text and further fine-tuned on targeted downstream task tables to achieve higher performance, improve the robustness of the BERT model, and thus improve the output efficiency and accuracy of the first and second results.

[0180] In summary, the intelligent question-answering device based on the BERT model described in this embodiment, by inputting the third word embedding vector into a pre-trained BERT model, outputs the probability value of each aggregation operation and the probability value of each cell in the associated table. The BERT model introduces targeted architectural adjustments based on the characteristics of tabular data, relies on a weakly supervised approach, directly obtains the final answer to the target question, and the model learns the aggregation operations and cell selections that may be involved. This is a more useful design pattern that not only solves the problem of manual costs but also makes the model more flexible, avoids the design of SQL rules, and improves the feedback efficiency of intelligent question-answering answers.

[0181] Example 3

[0182] See Figure 5 The diagram shown is a structural schematic of an electronic device provided in Embodiment 3 of this application. In a preferred embodiment of this application, the electronic device 3 includes a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0183] Those skilled in the art should understand that Figure 5 The structure of the electronic device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0184] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 3 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.

[0185] It should be noted that the electronic device 3 is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0186] In some embodiments, the memory 31 is used to store program code and various data, such as a BERT-based intelligent question-answering device 20 installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0187] In some embodiments, the at least one processor 32 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The at least one processor 32 is the control unit of the electronic device 3, connecting various components of the entire electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the electronic device 3.

[0188] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc.

[0189] Although not shown, the electronic device 3 may also include a power supply (such as a battery) to power the various components. Optionally, the power supply may be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power sources, a recharging device, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0190] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0191] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of this application.

[0192] In a further embodiment, combined with Figure 4 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed applications (such as the intelligent question-answering device 20 based on the BERT model), program code, etc., for example, the various modules mentioned above.

[0193] The memory 31 stores program code, and the at least one processor 32 can call the program code stored in the memory 31 to execute related functions. For example, Figure 4 The modules described herein are program codes stored in the memory 31 and executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of intelligent question answering based on the BERT model.

[0194] For example, the program code can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete this application. The one or more modules / units can be a series of computer-readable instruction segments capable of performing a specific function, which describe the execution process of the program code in the electronic device 3. For example, the program code can be divided into a first acquisition module 201, a second acquisition module 202, a third acquisition module 203, a determination module 204, an input module 205, and a fourth acquisition module 206.

[0195] In one embodiment of this application, the memory 31 stores a plurality of computer-readable instructions, which are executed by the at least one processor 32 to implement intelligent question answering based on the BERT model.

[0196] Specifically, the specific implementation method of the above instructions by the at least one processor 32 can be referred to Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0197] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0198] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0199] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0200] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or, and the singular does not exclude the plural. Multiple elements or devices recited in this application may also be implemented by a single element or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method for intelligent question answering based on a BERT model, characterized in that, The method includes: In response to a received intelligent question-answering request, obtain the target question that is identified by an aggregation operation and the associated table of the target question; Obtaining the first word embedding vector of the target question includes: performing word segmentation on the target question to obtain the word code of each word; obtaining the position code of each word in the target question; obtaining the sentence number code and rank sort code of each word; obtaining the column code and row code of each word; and superimposing the word code, the position code, the sentence number code, the column code, the row code and the rank sort code to form the first word embedding vector. Obtain the second word embedding vector of the association table; The first word embedding vector and the second word embedding vector are concatenated to obtain the third word embedding vector; The third word embedding vector is input into a pre-trained BERT model, which outputs the probability value of each aggregation operation and the probability value of each cell in the association table. The pre-training process of the BERT model includes: creating a pre-trained BERT model, which includes a word embedding layer, a position embedding layer, a sentence label embedding layer, a column index embedding layer, a row index embedding layer, and a rank-sorted embedding layer. The first label output position of the BERT model is connected to a softmax output layer; obtaining a set of historical questions with aggregation operation identifiers and an association table for each historical question from a preset data source; using the set of historical questions and the association table of the historical questions as a training set; and training the pre-trained BERT model based on the training set to obtain the BERT model. The answer to the target question is obtained based on the probability values ​​of multiple aggregation operations and multiple cells.

2. The intelligent question answering method based on the BERT model according to claim 1, wherein, The step of obtaining the second word embedding vector of the association table includes: The associated table is divided into header text and table text; The header text and table text are segmented into words to obtain the word code, position code, sentence number code, column code, row code, and rank sort code for each word. The first sub-embedding vector is formed by superimposing the word code, position code, sentence number code, column code, row code, and rank sort code corresponding to the header text; The word code, position code, sentence number code, column code, row code, and rank-sorted code corresponding to the table text are superimposed to form a second sub-embedding vector; The first sub-embedding vector and the second sub-embedding vector are concatenated to obtain the second word embedding vector.

3. The intelligent question answering method based on the BERT model according to claim 1, wherein, The step of using the historical problem set and its association table as a training set includes: Retrieve each historical question and its associated table; A preset number of word fragments are randomly selected from each historical question, and tokenized text is generated based on the preset number of word fragments; Extract the first character from the association table corresponding to the historical issues; Following the extraction order, the first word of each table is added to the end of the tokenized text; If the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, the addition stops, and the word fragment sequence corresponding to the historical question is obtained. If the length of the combined text of the tokenized text and the added table units does not meet the preset word fragment sequence length threshold, the second character of each table is extracted from the association table, and the above addition process is repeated until the length of the combined text of the tokenized text and the added table units meets the preset word fragment sequence length threshold, thus obtaining the word fragment sequence of the corresponding historical question. The set of word fragment sequences from the historical question set is used as the training set.

4. The intelligent question answering method based on the BERT model according to claim 1, wherein, The step of obtaining the answer to the target question based on the probability values ​​of multiple aggregation operations and multiple cell probabilities includes: Select the aggregation operation with the highest probability from the probability values ​​of the plurality of aggregation operations; The probability of each cell is compared with a preset probability threshold; Filter out the text in cells whose probability is greater than or equal to the preset probability threshold; Perform the corresponding aggregation operation on the text to obtain the answer to the target question.

5. The intelligent question answering method based on the BERT model according to claim 1, wherein, After obtaining the target problem identified by the aggregation operation and the association table of the target problem, the method further includes: In response to the intelligent question-answering request, obtain message information; Obtain the problem text from the message information; Insert a first marker at the beginning of the question text and a second marker at the end of the question text; The marked question text is identified as the target question.

6. An intelligent question answering device based on a BERT model, characterized in that, The apparatus is used to implement the intelligent question answering method based on the BERT model as described in any one of claims 1 to 5, and the apparatus comprises: The first acquisition module is used to, in response to a received intelligent question-answering request, acquire the target question identified by the aggregation operation and the association table of the target question; The second acquisition module is used to acquire the first word embedding vector of the target question; The third acquisition module is used to acquire the second word embedding vector of the association table; The determining module is used to concatenate the first word embedding vector and the second word embedding vector to obtain the third word embedding vector; The input module is used to input the third word embedding vector into the pre-trained BERT model and output the probability value of each aggregation operation and the probability value of each cell of the association table. The fourth acquisition module is used to obtain the answer to the target question based on the probability values ​​of multiple aggregation operations and the probability values ​​of multiple cells.

7. An electronic device, comprising: The electronic device includes a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the intelligent question answering method based on the BERT model as described in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by the processor, it implements the intelligent question-answering method based on the BERT model as described in any one of claims 1 to 5.

Images