Intelligent question and answer method, device and equipment and storage medium

Abstract

The embodiment of the invention provides an intelligent question answering method and device, equipment and a storage medium, and the method comprises the steps: carrying out the word segmentation of a to-be-answered text, and obtaining a serial number vector corresponding to the to-be-answered text; the sequence number vector corresponding to the to-be-answered text is input into an intelligent question and answer model, a reply text is obtained, the intelligent question and answer model is obtained by training question samples, and the intelligent question and answer model comprises an embedded layer, a self-attention layer and a decoding layer; the self-attention calculation mode of the self-attention layer is determined according to the output representation of the embedded layer on the serial number vector corresponding to the problem sample and the output representation of the decoding layer on the serial number vector of the problem sample. According to the embodiment of the invention, the information of the embedded layer and the decoding layer is introduced into the self-attention calculation, so that the information exchange between layers is enhanced, and the intelligent question and answer model can more comprehensively understand the context information, so that the intelligent question and answer model can more deeply understand the context, and the user experience is improved. Therefore, the accuracy and coherence of replying the text are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent question-answering method, device, equipment and storage medium. Background Art

[0002] Natural Language Processing (NLP) is a discipline that uses computers to understand human language. It covers multiple aspects from text analysis, information extraction to language generation. It aims to enable machines to understand, interpret and generate human language, thereby achieving natural and efficient human-computer interaction.

[0003] By applying advanced technologies such as machine learning and deep learning, NLP continues to promote the development of the field of artificial intelligence and provides strong technical support for many application scenarios such as intelligent customer service, sentiment analysis, machine translation, automatic summarization, and intelligent question and answer.

[0004] In the past few years, the Transformer model has become a hot topic in the field of natural language processing, and the Transformer deep learning model architecture has been widely used and evolved in almost all possible fields. This model not only performs well in natural language processing tasks, but also has great help and potential for other fields, especially time series prediction.

[0005] Traditional Transformer models still face certain limitations when processing complex and dynamic text data, especially in scenarios that require in-depth understanding of text context and precise expression of semantic information. Summary of the invention

[0006] The embodiments of the present invention provide an intelligent question-answering method, apparatus, device and storage medium, which solve the problem of low accuracy in text context understanding and semantic expression in traditional methods, and improve the accuracy of the intelligent question-answering method.

[0007] In a first aspect, an embodiment of the present invention provides an intelligent question-answering method, the method comprising:

[0008] Perform word segmentation on the text to be answered to obtain a sequence vector corresponding to the text to be answered;

[0009] The serial number vector corresponding to the text to be answered is input into the intelligent question and answer model to obtain a reply text, wherein the intelligent question and answer model is trained by question samples, and the intelligent question and answer model includes an embedding layer, a self-attention layer and a decoding layer, and the self-attention calculation method of the self-attention layer is determined according to the output representation of the serial number vector corresponding to the question sample by the embedding layer and the output representation of the serial number vector of the question sample by the decoding layer.

[0010] Furthermore, the self-attention layer includes an attention sublayer and a hidden state layer, and the intelligent question-answering model is trained by question samples and obtained through the following steps:

[0011] For the current iteration, obtaining an embedding feature according to the sequence vector corresponding to the problem sample and the embedding layer;

[0012] According to the embedded features, based on the output representation of the sequence vector corresponding to the problem sample by the embedding layer, the output representation of the sequence vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix, obtaining the attention features output by the attention sublayer;

[0013] According to the sequence vector corresponding to the question sample and the attention feature, based on the hidden state matrix corresponding to the current iteration and the first preset hidden state matrix, the hidden state feature output by the hidden state layer is obtained, and the hidden state matrix is ​​determined according to the dependency relationship between adjacent network layers in the intelligent question answering model;

[0014] Obtaining a response sample according to the latent feature and the decoding layer;

[0015] According to the attention feature, the second preset latent matrix and the representation of the sequence vector corresponding to the problem sample by the latent matrix, the latent matrix is ​​updated, and the next iteration is used as the current iteration again;

[0016] Repeat the above steps until the iteration end condition is met to obtain the intelligent question answering model.

[0017] Further, the step of obtaining the attention feature output by the attention sublayer based on the embedding feature, the output representation of the sequence vector corresponding to the problem sample by the embedding layer, the output representation of the sequence vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix comprises:

[0018] Obtaining a first attention feature according to the embedded feature, an output representation of the sequence vector corresponding to the problem sample by the embedding layer, and the first preset attention matrix;

[0019] Obtaining a second attention feature according to the embedded feature, the output representation of the sequence vector of the problem sample by the decoding layer, and the second preset attention matrix;

[0020] According to the first attention feature and the second attention feature, the attention feature output by the attention sub-layer is obtained.

[0021] Furthermore, the first attention feature is obtained according to the embedding feature, the output representation of the sequence vector corresponding to the problem sample by the embedding layer, and the first preset attention matrix, and is calculated by the following formula:

[0022] H embedding =E embedding (X);

[0023] Q=H i-1 *W Q ;

[0024] K1=H embedding *W K ;

[0025]

[0026] H″=H embedding *A1;

[0027] Where H″ represents the first attention feature, X represents the sequence vector corresponding to the problem sample, and E embedding represents the embedding layer, H embedding represents the output representation of the sequence vector corresponding to the problem sample by the embedding layer, H i-1 represents the input features of the i-th attention sublayer, W Q represents the first preset attention matrix, W K Denotes the second preset attention matrix, D K1 Represents the dimension of K1.

[0028] Furthermore, the second attention feature is obtained according to the embedded feature, the output representation of the sequence vector of the problem sample by the decoding layer and the second preset attention matrix, and is calculated by the following formula:

[0029] H decode =E decode (X);

[0030] Q=H i-1 *W Q ;

[0031] K2=H decode *W K ;

[0032]

[0033] H′=H decode *A2;

[0034] Among them, H′ represents the second attention feature, X represents the sequence vector corresponding to the problem sample, and E decoderepresents the decoding layer, H decode represents the output representation of the sequence vector corresponding to the problem sample by the decoding layer, H i-1 represents the input features of the i-th attention sublayer, W Q represents the first preset attention matrix, W K Denotes the second preset attention matrix, D K2 Represents the dimension of K2.

[0035] Further, the step of obtaining the hidden state features output by the hidden state layer according to the sequence vector corresponding to the problem sample and the attention feature, based on the hidden state matrix corresponding to the current iteration and the first preset hidden state matrix, comprises:

[0036] H hidden =W hidden (X);

[0037] Q=H att *W J ;

[0038]

[0039] H i =H hidden *A3;

[0040] Among them, H i represents the input features of the i+1th attention sublayer, H hidden represents the representation of the sequence vector corresponding to the problem sample taken out from the hidden state matrix, W hidden represents the hidden state matrix, X represents the sequence vector corresponding to the problem sample, H att represents the attention feature, W J represents the first preset latent matrix, D Hhidden Indicates H hidden Dimension.

[0041] Furthermore, the step of updating the hidden state matrix according to the attention feature, the second preset hidden state matrix and the characterization of the sequence vector corresponding to the problem sample by the hidden state matrix comprises:

[0042] I = sigmoid(H att *W I );

[0043] W hidden (X) = I*H att +H hidden ;

[0044] Among them, H att represents the attention feature, WI represents the second preset latent matrix, H hidden represents the representation of the sequence vector corresponding to the problem sample taken out from the hidden state matrix, W hidden Represents the hidden state matrix, and X represents the sequence number vector corresponding to the problem sample.

[0045] In a second aspect, an embodiment of the present invention provides an intelligent question-answering device, the device comprising:

[0046] A word segmentation module is used to perform word segmentation processing on the text to be answered and obtain a sequence vector corresponding to the text to be answered;

[0047] A question-answering module is used to input the serial number vector corresponding to the text to be answered into the intelligent question-answering model to obtain a reply text, wherein the intelligent question-answering model is trained by question samples, and the intelligent question-answering model includes an embedding layer, a self-attention layer and a decoding layer. The self-attention calculation method of the self-attention layer is determined according to the output representation of the serial number vector corresponding to the question sample by the embedding layer and the output representation of the serial number vector of the question sample by the decoding layer.

[0048] In a third aspect, an embodiment of the present invention provides a computer device, the device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0049] Memory, used to store computer programs;

[0050] The processor is used to implement the steps of an intelligent question-answering method provided in the first aspect when executing a program stored in the memory.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of an intelligent question-answering method provided in the first aspect are implemented.

[0052] The embodiment of the present invention provides an intelligent question-answering method, device, equipment and storage medium. When intelligent question-answering is required, the text to be answered is first segmented to obtain the sequence vector corresponding to the text to be answered; then the sequence vector is input into the intelligent question-answering model to obtain the reply text. The self-attention calculation method of the intelligent question-answering model in the embodiment of the present invention is determined by the output representation of the sequence vector corresponding to the question sample by the embedding layer and the output representation of the sequence vector of the sample question by the decoding layer. By introducing the information of the embedding layer and the decoding layer into the self-attention calculation, not only the information exchange between the layers is strengthened, but also the intelligent question-answering model can understand the context information more comprehensively. Therefore, the intelligent question-answering model can understand the context more deeply, thereby improving the accuracy and coherence of the reply text. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0054] Figure 1 A flowchart of an intelligent question-answering method provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of an intelligent question-answering device provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] In order to make the description of the disclosed content more detailed and complete, the following is an illustrative description of the implementation mode and specific examples of the present invention; however, this is not the only form of implementing or applying the specific embodiments of the present invention. The implementation mode covers the features of multiple specific embodiments and the method steps and their sequence for constructing and operating these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and step sequences. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0060] In the description of the embodiments of the present invention, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two, and other quantifiers are similar. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention, and the embodiments of the present application and the features in the embodiments may be combined with each other without conflict.

[0061] The embodiment of the present invention provides an intelligent question-answering method, which can be applied to an artificial intelligence question-answering client. When using the intelligent question-answering method, a text to be answered is inputted on the client, and the client sends the text to be answered to the server. After receiving the text to be answered, the server first performs word segmentation processing on the text to be answered to obtain a sequence vector corresponding to the text to be answered; then the sequence vector corresponding to the text to be answered is inputted into the intelligent question-answering model to obtain a reply text. The self-attention calculation method of the intelligent question-answering model in the embodiment of the present invention is determined by the output representation of the sequence vector corresponding to the question sample by the embedding layer and the output representation of the sequence vector of the sample question by the decoding layer. Since the embedding layer and the decoding layer are involved in the calculation process of the self-attention, the cross-layer interaction in the intelligent question-answering model is realized, so that the intelligent question-answering model can understand the context more deeply, thereby improving the accuracy of the reply text.

[0062] The server can be implemented by an independent server or a server cluster composed of multiple servers. The client can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The client and the server can be connected via Bluetooth, USB (Universal Serial Bus) or other communication connection methods, which is not limited in the embodiments of the present invention.

[0063] Figure 1 A flowchart of an intelligent question-answering method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0064] S110, performing word segmentation processing on the text to be answered to obtain a sequence vector corresponding to the text to be answered;

[0065] First, obtain the text to be answered, which is the text that needs to be answered, such as what happened today in history, etc. The specific method can be determined according to the actual situation, and the embodiment of the present invention does not make specific restrictions on this. Perform word segmentation on the text to be answered. Common word segmentation methods include rule-based word segmentation methods and statistical-based word segmentation methods. Common rule-based word segmentation methods include maximum matching methods and minimum matching methods. Common statistical-based word segmentation methods include n-gram grammar models and hidden Markov models. The specific method can be determined according to the actual situation, and the embodiment of the present invention does not make specific restrictions on this.

[0066] After word segmentation, the serial number corresponding to each word segment is obtained, and these serial numbers are combined into a vector as the serial number vector corresponding to the text to be answered. For example, for the text to be answered "What happened on this day in history", after word segmentation, "historically", "of", "today", "happened", "what" are obtained; for the word "historically", if it is in a certain vocabulary, it can be used as an independent vocabulary unit, and the index number of the vocabulary unit in the vocabulary is the serial number vector of the word segment; if it is in another vocabulary, "historically" may be further decomposed. For example, in a character-based tagging system, "history", "history" and "on" are separate index numbers, and the index of these three characters in the character set is the serial number vector corresponding to the word "historically". Other word segments obtain corresponding serial number vectors in the same way.

[0067] S120, inputting the serial number vector corresponding to the text to be answered into the intelligent question and answer model to obtain a reply text, wherein the intelligent question and answer model is trained by question samples, and the intelligent question and answer model includes an embedding layer, a self-attention layer and a decoding layer, and the self-attention calculation method of the self-attention layer is determined according to the output representation of the serial number vector corresponding to the question sample by the embedding layer and the output representation of the serial number vector of the question sample by the decoding layer.

[0068] The serial vector of the text to be answered is input into the intelligent question-answering model to obtain the reply text. The intelligent question-answering model is a neural network model. Before using the intelligent question-answering model, it needs to be trained with question samples. The intelligent question-answering model includes an embedding layer, a self-attention layer, and a decoding layer in sequence. In addition, during the training process, the self-attention calculation method of the self-attention layer can be determined according to the output representation of the serial vector of the question sample by the embedding layer and the output representation of the serial vector of the question sample by the decoding layer. The output representation of the serial vector of the question sample by the embedding layer refers to the output feature of the embedding layer after the serial vector is input into the embedding layer, and the output representation of the serial vector of the question sample by the decoding layer refers to the output feature after the serial vector is input into the decoding layer.

[0069] The intelligent question-answering model may be a Transformer model, which may be determined according to actual conditions, and the embodiments of the present invention do not specifically limit this. The question samples are generally extracted from references such as general fields, specific fields, dialogues and question-answering, news and social media, and may be determined according to actual conditions, and the embodiments of the present invention do not specifically limit this.

[0070] In addition, the embedding layer is mainly used to convert the input sequence vector into a low-dimensional continuous vector representation. Taking natural language processing as an example, the number of words in the vocabulary may be very large. If each word is represented by one-hot encoding, the vector will be very sparse and high in dimension. The embedding layer maps the words to a low-dimensional, dense vector space through learning. For example, a word in a vocabulary of 10,000 words is mapped to a 100-dimensional vector space.

[0071] In the intelligent question-answering model, the embedding layer is a very important first step. For the words in the question and answer text, the embedding layer can convert them into vectors, which can be used as input for subsequent neural network layers. This allows the model to better understand the semantic relationship between words and improve its ability to understand questions and answers. For example, when dealing with the question "What is the capital of China?", the embedding layer converts each word into a vector, allowing the model to analyze the semantic structure of the question through these vectors, providing a basis for subsequent answer generation or retrieval.

[0072] The embodiment of the present invention provides an intelligent question-answering method. When intelligent question-answering is required, the text to be answered is first segmented to obtain a sequence vector corresponding to the text to be answered; then the sequence vector is input into the intelligent question-answering model to obtain a reply text. The self-attention calculation method of the intelligent question-answering model in the embodiment of the present invention is determined by the output representation of the sequence vector corresponding to the question sample by the embedding layer and the output representation of the sequence vector of the sample question by the decoding layer. By introducing the information of the embedding layer and the decoding layer into the self-attention calculation, not only the information exchange between the layers is strengthened, but also the intelligent question-answering model can understand the context information more comprehensively. Therefore, the intelligent question-answering model can understand the context more deeply, thereby improving the accuracy and coherence of the reply text.

[0073] In some embodiments, the self-attention layer includes an attention sublayer and a hidden state layer, and the intelligent question-answering model is trained by question samples and obtained by the following steps:

[0074] For the current iteration, obtaining an embedding feature according to the sequence vector corresponding to the problem sample and the embedding layer;

[0075] According to the embedded features, based on the output representation of the sequence vector corresponding to the problem sample by the embedding layer, the output representation of the sequence vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix, obtaining the attention features output by the attention sublayer;

[0076] According to the sequence vector corresponding to the question sample and the attention feature, based on the hidden state matrix corresponding to the current iteration and the first preset hidden state matrix, the hidden state feature output by the hidden state layer is obtained, and the hidden state matrix is ​​determined according to the dependency relationship between adjacent network layers in the intelligent question answering model;

[0077] Obtaining a response sample according to the latent feature and the decoding layer;

[0078] According to the attention feature, the second preset latent matrix and the representation of the sequence vector corresponding to the problem sample by the latent matrix, the latent matrix is ​​updated, and the next iteration is used as the current iteration again;

[0079] Repeat the above steps until the iteration end condition is met to obtain the intelligent question answering model.

[0080] Different from the traditional Transformer model, the Transformer model in the embodiment of the present invention includes an embedding layer, a self-attention layer, and a decoding layer, and the self-attention layer includes an attention sublayer and a hidden state layer. The hidden state layer is used to replace the feedforward neural network in the original Transformer model. Therefore, when training the intelligent question-answering model, the training steps are as follows:

[0081] In the current iteration of the training, the sequence vector corresponding to the problem sample is input into the embedding layer to obtain the embedding vector. Here, the sequence vector of the problem sample is X = [x1, x2, ..., x L ], where x i represents the i-th sequence number, and L is the length of X. After X is input into the embedding layer, we get H0 = [h1,h2,…,h L ], with dimensions L*Z. The parameters of the embedding layer are denoted by E embedding Represents a size of V*Z, where V is the vocabulary size and Z is the vector length of each sequence number; the self-attention layer consists of N attention sub-layers, each of which implements the same calculation.

[0082] In the self-attention calculation, based on the embedded features, the output representation of the serial vector corresponding to the problem sample by the embedding layer, the output representation of the serial vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix, after a series of mathematical operations, the attention features output by the attention sub-layer can be obtained.

[0083] In the self-attention calculation layer of the traditional Transformer model, each sublayer includes two parts: self-attention calculation and feedforward neural network calculation. The output of the previous layer is used as the input of the next layer. Let the input of the i-th layer be H i-1 , the output is H i , the original self-attention calculation converts H i-1 Through three matrices W Q , W K , W V The self-attention operation is performed. However, the self-attention calculation process of the traditional Transformer model is the process of adjusting the vector representation. The original calculation method only involves the three matrices Q, K, and V, which is too dependent on the matrix W. Q , W K , W V , which makes the traditional Transformer model face certain limitations in scenarios that require in-depth understanding of text context and precise expression of semantic information.

[0084] Therefore, the present invention proposes a self-attention calculation method that introduces information from the embedding layer and the decoding layer. By introducing the information from the embedding layer and the decoding layer into the self-attention calculation, it not only strengthens the information exchange between layers, but also enables the intelligent question-answering model to understand the context information more comprehensively. This is especially important when processing long texts, dialogue generation, or machine translation tasks, because these tasks require the intelligent question-answering model to accurately understand and generate coherent text.

[0085] After obtaining the attention feature, the hidden state feature of the hidden state layer output is obtained based on the sequence vector corresponding to the problem sample and the attention feature, the hidden state matrix of the current iteration and the first preset hidden state matrix.

[0086] From the perspective of the input sequence, the calculations between different sequence numbers do not interfere with each other, so there is no hidden state layer and hidden state matrix in the traditional Transformer model. However, the Transformer model is composed of different network layers connected in series, and the output of the previous network layer is used as the input of the next network layer. There is a sequence dependency between layers, so a hidden state is needed to represent the reasoning state of the network layer. Therefore, in the embodiment of the present invention, a hidden state layer is set to replace the feedforward neural network, and a hidden state matrix is ​​used to represent the dependency between adjacent network layers in the intelligent question-answering model.

[0087] The present invention replaces the original feedforward neural network layer with a hidden state layer. This improvement is critical when dealing with tasks that require long-term memory and context understanding, such as intelligent question-answering systems or story generation. By retaining the dependencies between adjacent network layers in the intelligent question-answering model, the present invention not only enhances the encoding capability of the intelligent question-answering model, but also enables the intelligent question-answering model to maintain higher accuracy and coherence when processing long sequences and complex texts.

[0088] Finally, the latent features are input into the decoding layer to obtain the answer sample. In the embodiment of the present invention, the decoding layer parameters are E decode Indicates that the size is related to E embedding same.

[0089] In addition, according to the attention features, the second preset latent matrix and the representation of the serial number vector corresponding to the question sample by the latent matrix, the latent state matrix is ​​updated. After the update, it is determined whether the current iteration is the last iteration. If so, the training is terminated to obtain the trained intelligent question and answer model; if not, the next iteration is reused as the current iteration; the above steps are repeated until the iteration end condition is met to obtain the intelligent question and answer model.

[0090] As an implementation manner, the step of obtaining the attention feature output by the attention sublayer based on the embedding feature, the output representation of the sequence vector corresponding to the problem sample by the embedding layer, the output representation of the sequence vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix comprises:

[0091] Obtaining a first attention feature according to the embedded feature, an output representation of the sequence vector corresponding to the problem sample by the embedding layer, and the first preset attention matrix;

[0092] Obtaining a second attention feature according to the embedded feature, the output representation of the sequence vector of the problem sample by the decoding layer, and the second preset attention matrix;

[0093] According to the first attention feature and the second attention feature, the attention feature output by the attention sub-layer is obtained.

[0094] In the embodiment of the present invention, the first attention feature is obtained based on the embedding feature, the output representation of the sequence vector of the problem sample by the embedding layer, and the first preset attention matrix. The calculation formula is as follows:

[0095] H embedding =E embedding (X);

[0096] Q=H i-1 *W Q ;

[0097] K1=H embedding *W K ;

[0098]

[0099] H″=H embedding *A1;

[0100] Where H″ represents the first attention feature, X represents the sequence vector corresponding to the problem sample, and E embedding represents the embedding layer, H embedding represents the output representation of the sequence vector corresponding to the problem sample by the embedding layer, H i-1 represents the input features of the i-th attention sublayer, W Q represents the first preset attention matrix, W K Denotes the second preset attention matrix, D K1 Represents the dimension of K1.

[0101] Then, based on the embedded features, the output representation of the sequence vector of the problem sample by the decoding layer, and the second preset attention matrix, the second attention feature is obtained, which is calculated as follows:

[0102] H decode =E decode (X);

[0103] Q=H i-1 *W Q ;

[0104] K2=H decode *W K ;

[0105]

[0106] H′=H decode *A2;

[0107] Among them, H′ represents the second attention feature, X represents the sequence vector corresponding to the problem sample, and E decode represents the decoding layer, H decode represents the output representation of the sequence vector corresponding to the problem sample by the decoding layer, H i-1 represents the input features of the i-th attention sublayer, W Q represents the first preset attention matrix, W K Denotes the second preset attention matrix, D K2 Represents the dimension of K2.

[0108] Finally, the first attention feature and the second attention feature are summed to obtain the attention feature. The calculation formula is as follows:

[0109] H att =H″+H′.

[0110] It can be seen from the above formula that the embedding layer and the decoding layer participate in the attention machine calculation process, thereby realizing cross-layer information interaction, enabling the intelligent question-answering model to understand the context more deeply.

[0111] As an implementation manner, the step of obtaining the hidden state features output by the hidden state layer according to the sequence vector corresponding to the problem sample and the attention feature, based on the hidden state matrix corresponding to the current iteration and the first preset hidden state matrix, comprises:

[0112] H hidden =W hidden (X);

[0113] Q=H att *W J ;

[0114]

[0115] H i =H hidden *A3;

[0116] Among them, H i represents the input features of the i+1th attention sublayer, H hidden represents the representation of the sequence vector corresponding to the problem sample taken out from the hidden state matrix, W hidden represents the hidden state matrix, X represents the sequence vector corresponding to the problem sample, H att represents the attention feature, W J represents the first preset latent matrix, D Hhidden Indicates H hidden Dimension.

[0117] Considering that both the embedding layer and the decoding layer are V*Z matrices, the present invention uses a matrix W of size V*Z hidden As the hidden state of the network layer, the matrix can also be considered as a model's hidden state representation of the word vector. In addition, the present invention sets two matrices W of size Z*Z in this layer I , W J , these three matrices are used to replace the original feedforward neural network layer and perform attention calculations on the input and hidden states.

[0118] As an implementation manner, the step of updating the hidden state matrix according to the attention feature, the second preset hidden state matrix and the characterization of the sequence vector corresponding to the problem sample by the hidden state matrix comprises:

[0119] I = sigmoid(H att *W I );

[0120] W hidden (X) = I*H att +H hidden ;

[0121] Among them, H att represents the attention feature, W I represents the second preset latent matrix, H hidden represents the representation of the sequence vector corresponding to the problem sample taken out from the hidden state matrix, W hidden Represents the hidden state matrix, and X represents the sequence number vector corresponding to the problem sample.

[0122] In the embodiment of the present invention, I is a matrix whose elements are between 0 and 1, indicating H att Each element in is retained until W hidden Since the present invention aims to effectively express the hidden state information, only one hidden state matrix W is retained in the model. hidden (X), and the parameters of this matrix are shared among all layers. After the introduction of the hidden state matrix, the model can effectively handle scenarios such as long context and complex semantics, thereby enhancing the model's performance in scenarios such as question-answering systems.

[0123] Figure 2 A schematic diagram of the structure of an intelligent question-answering device provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the device comprises:

[0124] The word segmentation module 210 is used to perform word segmentation processing on the text to be answered to obtain a sequence vector corresponding to the text to be answered;

[0125] The question-and-answer module 220 is used to input the serial number vector corresponding to the text to be answered into the intelligent question-and-answer model to obtain a reply text, wherein the intelligent question-and-answer model is trained by question samples, and the intelligent question-and-answer model includes an embedding layer, a self-attention layer and a decoding layer. The self-attention calculation method of the self-attention layer is determined according to the output representation of the serial number vector corresponding to the question sample by the embedding layer and the output representation of the serial number vector of the question sample by the decoding layer.

[0126] This embodiment is a device embodiment corresponding to the above method embodiment, and its implementation process is the same as that of the above method embodiment. For details, please refer to the above method embodiment, and this device embodiment will not be repeated here.

[0127] Based on the above intelligent question-answering method, an embodiment of the present invention further provides a computer device, Figure 3A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the computer device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0128] Memory, used to store computer programs;

[0129] The processor is used to implement the steps of the above-mentioned task scheduling processing method when executing the program stored in the memory.

[0130] For other details about the processor in the above-mentioned computer device implementing the above-mentioned technical solution, please refer to the description of the task scheduling processing method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0131] Among them, the processor can also be called CPU (Central Processing Unit), the processor may be an integrated circuit chip with signal processing capabilities; the processor can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gata Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0132] The embodiment of the present invention also provides a computer-readable storage medium, on which a readable computer program is stored; wherein the computer program can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0134] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0135] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0136] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0137] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0138] The technical solution provided by the present application is introduced in detail above. The principles and implementation methods of the present application are explained by using specific examples in the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An intelligent question-answering method, characterized in that: The method comprises: Perform word segmentation on the text to be answered to obtain a sequence vector corresponding to the text to be answered; The serial number vector corresponding to the text to be answered is input into the intelligent question and answer model to obtain a reply text, wherein the intelligent question and answer model is trained by question samples, and the intelligent question and answer model includes an embedding layer, a self-attention layer and a decoding layer, and the self-attention calculation method of the self-attention layer is determined according to the output representation of the serial number vector corresponding to the question sample by the embedding layer and the output representation of the serial number vector of the question sample by the decoding layer.

2. The intelligent question-answering method according to claim 1, characterized in that: The self-attention layer includes an attention sublayer and a hidden state layer. The intelligent question-answering model is trained by question samples and obtained through the following steps: For the current iteration, obtaining an embedding feature according to the sequence vector corresponding to the problem sample and the embedding layer; According to the embedded features, based on the output representation of the sequence vector corresponding to the problem sample by the embedding layer, the output representation of the sequence vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix, obtaining the attention features output by the attention sublayer; According to the sequence vector corresponding to the question sample and the attention feature, based on the hidden state matrix corresponding to the current iteration and the first preset hidden state matrix, the hidden state feature output by the hidden state layer is obtained, and the hidden state matrix is ​​determined according to the dependency relationship between adjacent network layers in the intelligent question answering model; Obtaining a response sample according to the latent feature and the decoding layer; According to the attention feature, the second preset latent matrix and the representation of the sequence vector corresponding to the problem sample by the latent matrix, the latent matrix is ​​updated, and the next iteration is used as the current iteration again; Repeat the above steps until the iteration end condition is met to obtain the intelligent question answering model.

3. The intelligent question-answering method according to claim 2, characterized in that: The step of obtaining the attention feature output by the attention sublayer based on the output representation of the sequence vector corresponding to the problem sample by the embedding layer, the output representation of the sequence vector of the problem sample by the decoding layer, the first preset attention matrix and the second preset attention matrix according to the embedding feature comprises: Obtaining a first attention feature according to the embedded feature, an output representation of the sequence vector corresponding to the problem sample by the embedding layer, and the first preset attention matrix; Obtaining a second attention feature according to the embedded feature, the output representation of the sequence vector of the problem sample by the decoding layer, and the second preset attention matrix; According to the first attention feature and the second attention feature, the attention feature output by the attention sub-layer is obtained.

4. The intelligent question-answering method according to claim 3, characterized in that: The first attention feature is obtained according to the embedding feature, the output representation of the sequence vector corresponding to the problem sample by the embedding layer, and the first preset attention matrix, and is calculated by the following formula: H embedding =E embedding (X); Q=H i-1 *W Q ; K1=H embedding *IN K ; H″=H embedding *A1; Where H″ represents the first attention feature, X represents the sequence vector corresponding to the problem sample, and E embedding represents the embedding layer, H embedding represents the output representation of the sequence vector corresponding to the problem sample by the embedding layer, H i-1 represents the input features of the i-th attention sublayer, W Q represents the first preset attention matrix, W K Denotes the second preset attention matrix, D K1 Represents the dimension of K1.

5. The intelligent question-answering method according to claim 3, characterized in that: The second attention feature is obtained according to the embedded feature, the output representation of the sequence vector of the problem sample by the decoding layer and the second preset attention matrix, and is calculated by the following formula: H decode =E decode (X); Q=H i-1 *W Q ; K2=H decode *W K ; H′=H decode *A2; Among them, H′ represents the second attention feature, X represents the sequence vector corresponding to the problem sample, and E decode represents the decoding layer, H decode represents the output representation of the sequence vector corresponding to the problem sample by the decoding layer, H i-1 represents the input features of the i-th attention sublayer, W Q represents the first preset attention matrix, W K Denotes the second preset attention matrix, D K2 Represents the dimension of K2.

6. The intelligent question-answering method according to claim 2, characterized in that: The step of obtaining the hidden state features output by the hidden state layer according to the sequence vector corresponding to the problem sample and the attention feature and based on the hidden state matrix corresponding to the current iteration and the first preset hidden state matrix comprises: H hidden =W hidden (X); Q=H att *W J ; <h2 style=";text-align:left;direction:ltr">H<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> =H<h2 style=";text-align:left;direction:ltr"> hidden <h2 style=";text-align:left;direction:ltr"> *A3; Among them, H i represents the input features of the i+1th attention sublayer, H hidden represents the representation of the sequence vector corresponding to the problem sample taken out from the hidden state matrix, W hidden represents the hidden state matrix, X represents the sequence vector corresponding to the problem sample, H att represents the attention feature, W J represents the first preset latent matrix, D Hhidden Indicates H hidden Dimension.

7. The intelligent question-answering method according to claim 2, characterized in that: The step of updating the hidden state matrix according to the attention feature, the second preset hidden state matrix and the characterization of the sequence vector corresponding to the problem sample by the hidden state matrix comprises: I=sigmoid(H att *W I ); W hidden (X)=I*H att +H hidden ; Among them, H att represents the attention feature, W I represents the second preset latent matrix, H hidden represents the representation of the sequence vector corresponding to the problem sample taken out from the hidden state matrix, W hidden Represents the hidden state matrix, and X represents the sequence number vector corresponding to the problem sample.

8. An intelligent question-answering device, characterized in that: The device comprises: A word segmentation module is used to perform word segmentation processing on the text to be answered and obtain a sequence vector corresponding to the text to be answered; A question-answering module is used to input the serial number vector corresponding to the text to be answered into the intelligent question-answering model to obtain a reply text, wherein the intelligent question-answering model is trained by question samples, and the intelligent question-answering model includes an embedding layer, a self-attention layer and a decoding layer. The self-attention calculation method of the self-attention layer is determined according to the output representation of the serial number vector corresponding to the question sample by the embedding layer and the output representation of the serial number vector of the question sample by the decoding layer.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the intelligent question-answering method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent question-answering method according to any one of claims 1 to 7 are implemented.

Images