Method and device for analyzing text information using attention model, and electronic device

Abstract

The present disclosure provides a method, device, and electronic device for analyzing text information using an attention model, which can be applied to artificial intelligence, finance, and other fields. The attention model includes an input layer, a context encoding layer, an information interaction layer, an information reasoning layer, and an output layer. The method includes: using the input layer to convert the question word and the context sequence included in the input text information into a question vector Q and a context vector P respectively; and using the context encoding layer to encode and extract features of the question vector and the context vector respectively to obtain encoded question features and encoded context features; using the information interaction layer to fuse the encoded question feature U and the encoded context feature H to obtain a context feature Z for the question word; using the information reasoning layer to obtain an encoded context paragraph feature M for the context feature Z; and using the output layer to process the encoded context paragraph feature to obtain and output the position of the question word in the context sequence.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and more specifically, to a method, device, electronic device, computer-readable storage medium, and computer program product for analyzing text information using an attention model. Background Art

[0002] With the development of deep learning technology, the attention mechanism has become the mainstream method for designing machine reading comprehension models. In the attention model of related technologies, the attention mechanism is used to calculate the similarity between context and question pairs, and to construct feature representations of context and paragraphs based on the similarity matrix, so as to achieve semantic interaction between context and question, and effectively improve the text representation ability of the model. However, for these model architectures based on the attention mechanism, the multi-layer attention operations are relatively independent, and the operation of each layer of attention has not taken into account the influence of the previous layer of attention. Therefore, in the multi-layer attention framework, the distribution of multiple layers of attention on the same text will lead to the phenomenon of attention redundancy. In addition, the multi-layer attention fails to focus on the important content of the text, resulting in the problem of attention loss of important information. Summary of the invention

[0003] In view of this, the present disclosure provides a method and device, an electronic device and a computer-readable storage medium for analyzing text information using an attention model. Based on the question words and context sequences of the input text information, a multi-layer attention model is used to realize information interaction between the question words and the context sequences, and the position of the context sequence where the question words are located is obtained and output.

[0004] One aspect of the present disclosure provides a method for analyzing text information using an attention model, wherein the attention model includes an input layer, a context encoding layer, an information interaction layer, an information reasoning layer, and an output layer, wherein the method includes: using the input layer to convert a question word and a context sequence included in the input text information into a question vector Q and a context vector P, respectively; and using the context encoding layer to encode and extract features from the question vector Q and the context vector P, respectively, to obtain an encoded question feature U and an encoded context feature H; using the information interaction layer to fuse the encoded question feature U and the encoded context feature H, to obtain a context feature Z for the question word; using the information reasoning layer to obtain an encoded context paragraph feature M for the context feature Z; and using the output layer to process the encoded context paragraph feature M to obtain and output the position of the question word in the context sequence.

[0005] According to an embodiment of the present disclosure, the context encoding layer includes multiple bidirectional Transformer modules, each of which includes a self-attention module, a feedforward neural network and an output module.

[0006] According to an embodiment of the present disclosure, the information interaction layer includes a stacked multi-layer attention module, each layer of the attention module includes a question and context alignment module, a context self-alignment module and a context reasoning module; the use of the information interaction layer to fuse the encoded question feature U and the encoded context feature H to obtain the context feature Z for the question word includes: using the question and context alignment module to obtain a context paragraph representation V for the question word according to the encoded question feature U and the encoded context feature H; using the context self-alignment module to obtain a feature representation of each word in the context sequence according to the encoded context feature H and the context paragraph representation V And using the contextual reasoning module, according to the feature representation of each word A context feature representation Z for the question word is obtained.

[0007] According to an embodiment of the present disclosure, the use of the question and context alignment module to obtain the context paragraph representation V for the question word according to the encoded question feature U and the encoded context feature H includes: calculating a first similarity matrix E between the encoded question feature U and the encoded context feature H; calculating the attention distribution Softmax(E) of each word in the context sequence with respect to the question word according to the similarity matrix E; and calculating the context paragraph representation V for the question word according to the attention distribution Softmax(E).

[0008] According to an embodiment of the present disclosure, the calculating of the context paragraph representation V for the question word according to the attention distribution includes: calculating the context paragraph representation matrix of each word in the context sequence with respect to the question word according to the encoded question feature U and the attention distribution Softmax(E) By representing the context paragraph matrix The elements in are weighted and summed to calculate the context paragraph representation V for the question word.

[0009] According to an embodiment of the present disclosure, the feature representation of each word in the context sequence is calculated based on the encoded context feature H and the context paragraph representation V. The method comprises: calculating a similarity matrix S between each word in the context sequence; calculating the attention distribution Softmax(S) of each word in the context sequence with respect to the context sequence according to the similarity matrix S; and calculating the feature representation of each word in the context sequence according to the attention distribution Softmax(S).

[0010] According to an embodiment of the present disclosure, for the multi-layer attention module, according to the similarity matrix E of the j-1th layer attention j-1 And the similarity matrix S j-1 , calculate the similarity matrix E of the j-th layer attention module j And the similarity matrix S j j is greater than or equal to 1 and less than or equal to J, where J is the total number of layers of the attention module.

[0011] According to an embodiment of the present disclosure, the method further includes: using the position of the question word in the context sequence to determine at least one question corresponding to the question word, so as to provide an answer to the at least one question.

[0012] Another aspect of the present disclosure provides a device for analyzing text information using an attention model, wherein the attention model includes an input layer, a context encoding layer, an information interaction layer, an information reasoning layer, and an output layer, and the device includes:

[0013] A first module is used to convert the question words and context sequences included in the input text information into a question vector Q and a context vector P respectively using an input layer; and

[0014] The second module is used to use the context coding layer to encode and extract features from the question vector and the context vector respectively, so as to obtain a coded question feature and a coded context feature;

[0015] The third module is used to fuse the encoded question feature U and the encoded context feature H using the information interaction layer to obtain a context feature Z for the question word;

[0016] A fourth module is used to obtain an encoded context paragraph feature M for the context feature Z using the information inference layer; and

[0017] The fifth module is used to use the output layer to process the encoded context paragraph features to obtain and output the position of the question word in the context sequence.

[0018] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method described above.

[0019] Another aspect of the present disclosure provides a computer-readable storage medium having executable instructions stored thereon. When the instructions are executed by a processor, the processor is caused to perform the method described above.

[0020] Another aspect of the present disclosure provides a computer program product, wherein the product stores a computer program, and when the computer program is executed, the method described above can be implemented.

[0021] According to the embodiments of the present disclosure, the problems of lack of attention and redundancy in the reading comprehension model in the related art can be at least partially solved. The position of the content related to the question word generated according to the method in the context sequence is more accurate, and the method can extract text semantic features more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 The application scenario of the method and device for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 2A A flowchart of a method for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 2B The structural diagram of the attention model according to the embodiment of the present disclosure is schematically shown;

[0026] Figure 3A The structure diagram of the attention module in the attention model according to the embodiment of the present disclosure is schematically shown;

[0027] Figure 3B A flowchart of using the information interaction layer to fuse the coding problem feature and the coding context feature to obtain the context feature for the problem according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 3C A flowchart of obtaining a context paragraph representation for a question word in a method for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 3DSchematically shows a flow chart of calculating a context paragraph representation for a question word according to attention distribution according to an embodiment of the present disclosure;

[0030] Figure 3E A flowchart for calculating a feature representation of each word in a context sequence according to an encoded context feature and a context paragraph representation according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 4 The schematic diagram shows the structure of the bidirectional Transformer module in the attention model according to the embodiment of the present disclosure;

[0032] Figure 5 A block diagram of an apparatus for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0035] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0036] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0037] When using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of the said features.

[0038] The term "highway network" refers to an architectural design method for building deep neural networks, which can help build deeper neural networks without making the model difficult to train. The highway network uses gating functions to adjust the proportion of information flow, allowing information to flow in multi-layer networks without attenuation, while helping the model converge quickly.

[0039] The interaction process between customers and telephone banks can be converted from audio files to text files through ASR technology. Machine reading comprehension technology can be used to extract customer intentions from such texts and complete the collection and statistics of customer intentions. This is especially important for unconfigured conversation scenarios. Currently, for unrecognizable scenarios expressed by customers, the common practice is to broadcast a fallback script to remind customers to choose existing scenario conversations. Machine reading comprehension technology can extract conversation scenarios that users are interested in but have not yet realized. For example, through the audio content of customer feedback, the audio content is converted into a text file, and the customer's questions and context information are extracted through machine reading comprehension, and the text information for the questions is displayed to the user.

[0040] Embodiments of the present disclosure provide a method, device, electronic device, and medium for analyzing text information using an attention model. The question vector Q and context vector P are obtained by converting the question word and context sequence in the input text information, extracting the encoded question feature U and the encoded context feature H, and further obtaining the context feature Z for the question word, and finally obtaining the position of the question word in the context sequence. The method of the present disclosure solves the problems of lack of attention and redundancy in the reading comprehension model in the related art. The position of the content related to the question word generated according to the method in the context sequence is more accurate, and the method can more accurately extract text semantic features.

[0041] Figure 1 The exemplary system architecture 100 of the method for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0042] It should be noted that the method, device, electronic device and storage medium for analyzing text information using an attention model provided in the embodiments of the present disclosure can be used in the fields of artificial intelligence technology and finance, and can also be used in other fields outside the financial field, such as the field of machine learning technology. The method, device, electronic device and storage medium for analyzing text information using an attention model provided in the embodiments of the present disclosure do not limit the application field.

[0043] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0044] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0045] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0046] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0047] It should be noted that the method for analyzing text information using an attention model provided in the embodiment of the present disclosure can generally be performed by the server 105. Accordingly, the apparatus for analyzing text information using an attention model provided in the embodiment of the present disclosure can generally be set in the server 105. The method for analyzing text information using an attention model provided in the embodiment of the present disclosure can also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the apparatus for analyzing text information using an attention model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0049] Figure 2A The flowchart of the method for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown.

[0050] Figure 2B The structural diagram of the attention model according to the embodiment of the present disclosure is schematically shown. Figure 2B As shown, the attention model 200 includes an input layer 210, a context encoding layer 220, an information interaction layer 230, an information reasoning layer 240 and an output layer 250.

[0051] Combine the following Figure 2A and Figure 2B The method of analyzing text information using the attention model in an embodiment of the present disclosure is described in detail.

[0052] like Figure 2A As shown, the process of the method for analyzing text information using an attention model according to an embodiment of the present disclosure includes operations S201 to S205.

[0053] In operation S201 , question words and context sequences included in input text information are converted into a question vector Q and a context vector P, respectively, using the input layer 210 .

[0054] The question vector can be, for example, a word vector or a sentence vector. A word vector can convert words or phrases into a mathematical representation of a high-dimensional continuous vector space. Word vector representations include one-hot representation word vectors and distributed word vectors. Sentence vectors represent sentences of variable length with fixed-length vectors to provide services for NLP downstream tasks.

[0055] According to an embodiment of the present disclosure, by converting question words and context sequences in input text information into question vectors Q and context vectors P, context global information can be better considered and more effective semantic reasoning can be performed.

[0056] In the input layer 210, the text sequence is encoded from multiple granularities such as word vectors, character embedding vectors, position embedding vectors, and part-of-speech embedding vectors. First, the pre-trained 300-dimensional Glove word vectors are used to embed the input text sequence, and a fixed-dimensional vector representation is obtained for each word in the input text sequence. At the same time, a one-dimensional convolutional neural network (CNN) is used to encode the characters of the word, mapping each character into a high-dimensional vector, where the size of the character embedding vector dimension is the number of convolution kernel channels, and finally the input vector is max-pooled in width to obtain the character embedding vector of the word.

[0057] In addition, the absolute position embedding method and relative position embedding method used in the Transformer model are used to obtain the position embedding vector of the text sequence, and the part-of-speech sequence of the text is input into a one-layer convolutional neural network (CNN) by tagging the input text sequence, encoding the word's part-of-speech to obtain the part-of-speech embedding representation. Then, the position embedding vector, part-of-speech embedding vector and word vector of the sequence are added, and the word vector and character vector of the word are concatenated in the vector dimension, and finally the embedding representation X of the question and context paragraph is obtained. Q and X P ,

[0058]

[0059]

[0060] in d is the word embedding dimension.

[0061] After simple addition and concatenation of word vectors, the input text sequence is converted into a feature embedding representation containing semantic information. In order to better integrate these text feature information, the text feature representation is input into a two-layer high-speed network. After the high-speed network processes the text feature representation, the feature sequences of the question and the context paragraph are output respectively. and

[0062] In operation S202, the context coding layer 220 is used to encode and extract features from the question vector Q and the context vector P, respectively, to obtain encoded question features U and encoded context features H.

[0063] For example, a Transformer model based on the attention mechanism is used as a feature encoder. The text feature representation output by the high-speed network is input into an encoder composed of three layers of bidirectional Transformer modules, where each layer of the Transformer module consists of a self-attention module, a feedforward neural network, and an output module, and residual connections and layer normalization operations are performed on the output part of the module. The text features are contextually encoded through the three-layer bidirectional Transformer module, and finally the encoded question feature U and the encoded context feature H are obtained, where and

[0064] According to the embodiments of the present disclosure, by using the above-mentioned Transformer model as a feature encoder, long text sequences can be well processed and parallel operations can be performed. The Transformer module is introduced as a question and paragraph information encoder, which effectively improves the feature encoding ability of the model, and a pointer network is used as a candidate answer boundary predictor to return the final answer.

[0065] Figure 3A The structural diagram of the attention module in the attention model according to an embodiment of the present disclosure is schematically shown.

[0066] In operation S203, the information interaction layer 230 is used to fuse the encoded question feature U and the encoded context feature H to obtain a context feature Z for the question word.

[0067] For example, the information interaction layer 230 includes stacked multi-layer attention modules 231. For example, the number of layers of the attention module 231 can be three, four or five, etc. The specific number of layers can be set according to actual needs. The attention modules 231 are connected in series to form an information interaction layer.

[0068] In the embodiments of the present disclosure, Figure 3A As shown, each layer of attention module 231 may include a question-context alignment module 310, a context self-alignment module 320, and a context reasoning module 330. The input end of the question-context alignment module 310 is connected to the output end of the context encoding layer 220 to obtain the encoded question feature U and the encoded context feature H from the context encoding layer 220. The output end of the question-context alignment module 310 is connected to the input end of the context self-alignment module 320, and the output end of the context self-alignment module 320 is connected to the context reasoning module 330. In an embodiment of the present disclosure, the question-context alignment module 310, the context self-alignment module 320, and the context reasoning module 330 are connected in series.

[0069] In the embodiment of the present disclosure, the multi-layer attention module is composed of a plurality of single-layer attention modules 231 stacked together, and each single-layer attention module operation is connected using a multi-layer aligned attention mechanism, which is calculated according to formula (1):

[0070] For example, take the connection of the three-layer aligned attention layer as an example:

[0071] Z 1 , E 1 , S 1 =Align 1 (U, H)

[0072] Z 2 , E 2 , S 2 =Align 2 (U, H, Z 1 , E 1 , S 1 )

[0073] Z 3 , E 3 , S 3 =Align 3 (U, H, Z 2 , E 2 , S 2 ) (1)

[0074] Among them, Align l Represents the first layer of aligned attention layer, except for the similarity matrix E of the first layer 1 , S 1 In addition, the similarity matrices in the remaining layers are updated using formulas (2) and (3). At the same time, residual connections Z are used in each layer. l =Align l (Z l-1 )+Z l-1 Get the feature output of each layer.

[0075] Formula (2):

[0076]

[0077]

[0078] Where σ(·) is the sigmoid function, W g1 is a trainable parameter matrix.

[0079] Formula (3):

[0080]

[0081]

[0082] Among them, W g2 is the trainable parameter matrix, Context sequence feature representation output by the previous attention layer.

[0083] According to the embodiments of the present disclosure, in the multi-layer attention module, by aligning the question with the context sequence, the extracted high-dimensional features of the context sequence are also aligned with the context sequence features. In this way, the context sequence information can be continuously added to the multi-layer attention operation to avoid the loss of real context feature information in the deep attention operation.

[0084] Figure 3B The flowchart of using the information interaction layer to fuse the coding question feature and the coding context feature to obtain the context feature for the question according to an embodiment of the present disclosure is schematically shown.

[0085] In an embodiment of the present disclosure, the process of using the information interaction layer 230 to fuse the encoded question feature U and the encoded context feature H to obtain the context feature Z for the question includes operations S301 to S303.

[0086] In operation S301, the question and context alignment module 310 is used to obtain a context paragraph representation V for the question according to the encoded question feature U and the encoded context feature H.

[0087] In operation S302, the context self-alignment module 320 is used to obtain the feature representation of each word in the context sequence according to the encoded context feature H and the context paragraph representation V.

[0088] In operation S303, the context inference module 330 is used to calculate the feature representation of each word. Get the context feature representation Z for the problem.

[0089] In the embodiment of the present disclosure, the question and context alignment module 310 is used to process the information interaction between the question and the context paragraph, and obtain the context paragraph representation V for the question according to the encoded question feature U and the encoded context feature H.

[0090] Figure 3C The flowchart of the method for analyzing text information using an attention model according to an embodiment of the present disclosure for obtaining a context paragraph representation for a question word is schematically shown.

[0091] like Figure 3C As shown, the operation flow of obtaining the context paragraph representation V for the question includes operations S311 to S313.

[0092] In operation S311, a first similarity matrix E between the encoding question features U and the encoding context features H is calculated.

[0093] For example, given the encoding question feature U and the encoding context feature H, the formula is expressed as follows:

[0094] U={u1,u2…,u m}

[0095] H={h1,h2 . . . , h n}

[0096] Where m and n represent the length of the question word and the context paragraph respectively. The attention similarity between the encoding question feature U and the encoding context feature H The matrix can be calculated by formula (4):

[0097]

[0098] Among them, i∈[1,m], j∈[1,n], E ij represents the semantic similarity between the i-th word in the question sequence and the j-th word in the context paragraph, is the similarity function between words. In different models, the similarity function is also different.

[0099] Similarity calculation function The definition is shown in formula (5):

[0100]

[0101] Among them, ⊙ is the multiplication of vector elements, [;] is the concatenation operation between vectors, is the trainable weight matrix.

[0102] In operation S312, the attention distribution Softmax(E) of each word in the context sequence with respect to the question word is calculated according to the similarity matrix E.

[0103] For example, the attention distribution Softmax(E) can be calculated based on the number of question words in the context. For example, the attention distribution of the jth word in the context about the question can be obtained based on the similarity matrix

[0104] In operation S313, a context paragraph representation V for the question word is calculated according to the attention distribution Softmax (E).

[0105] Figure 3D A flowchart for calculating a context paragraph representation for a question word based on attention distribution according to an embodiment of the present disclosure is schematically shown.

[0106] like Figure 3D As shown, the flowchart for calculating the context paragraph representation V for the question word according to the attention distribution Softmax (E) includes operations S321 to S322.

[0107] In operation S321, the context paragraph representation matrix of each word in the context sequence with respect to the question word is calculated based on the encoded question feature U and the attention distribution Softmax(E).

[0108] For example, based on the similarity evidence, we can obtain the attention distribution of the jth word in the context about the question word. Get the context paragraph representation matrix of the jth word about the question word

[0109] In operation S322, by representing the context paragraph matrix The elements in are weighted and summed to calculate the context paragraph representation V for the question word.

[0110] For example, by creating an output gate through the highway network, the input ratio of the context feature is controlled, and the output feature is nonlinearly changed. The final context paragraph representation V for the question word output by the question and context alignment module is shown in formula (6):

[0111]

[0112]

[0113]

[0114] Among them, W r and is a trainable weight parameter, and the context paragraph for the question word is represented as

[0115] According to an embodiment of the present disclosure, by introducing a highway network into the question-context alignment module, it is possible to better control the input ratio of question-aware context paragraph features and the problem of gradient vanishing caused by multi-layer attention calculation.

[0116] In the embodiments of the present disclosure, Figure 3A As shown, the context self-alignment module 320 is used to perform self-attention operation on the context paragraphs, capture the semantic information between different layers of context paragraphs, and obtain the feature representation of each word in the context sequence.

[0117] Figure 3EThe flowchart of calculating the feature representation of each word in the context sequence according to the encoded context features and the context paragraph representation according to an embodiment of the present disclosure is schematically shown.

[0118] like Figure 3A and 3E As shown, using the context self-alignment module 320, the feature representation of each word in the context sequence is obtained according to the encoded context feature H and the context paragraph representation V The process includes operation S331 to operation S333.

[0119] In the context self-alignment module 320, the long-distance dependencies between different words in the context are captured by performing self-attention operation on the encoded context feature H.

[0120] In operation S331, a similarity matrix S between each word in the context sequence is calculated.

[0121] For example, using the initial encoded context feature H to perform self-attention operation, the similarity matrix S between each word in the context sequence is calculated according to formula (5) and formula (7),

[0122] Wherein formula (7) is:

[0123]

[0124] Among them, i∈[1,n], j∈[1,n], when i=j, it means that the context paragraph word calculates the similarity with itself, which will cause the similarity of the paragraph word to be concentrated on itself and cannot be extended to other words. Therefore, the similarity at this time is set to 0 to avoid the paragraph word not calculating the similarity with itself. When i≠j, it means the similarity between different words in the context paragraph, so the similarity calculation function is used Calculate the semantic similarity between different words.

[0125] The similarity matrix between each word in the calculated context sequence As shown in formula (8):

[0126]

[0127] Among them, W s is a trainable weight parameter matrix.

[0128] In operation S332, the attention distribution Softmax(S) of each word in the context sequence with respect to the context sequence is calculated according to the similarity matrix S.

[0129] For example, by performing Softmax normalization on the self-similarity matrix of the context sequence, we can obtain the attention distribution of each word in the context sequence about the context:

[0130] In operation S333, the feature representation of each word in the context sequence is calculated according to the attention distribution Softmax(S)

[0131] For example, use the attention distribution as the weight of the context sequence feature vector for weighted summation to obtain the weighted feature representation of each paragraph word As shown in formula (9):

[0132]

[0133] In the embodiment of the present disclosure, the context reasoning module 330 is used to further reason about the context paragraph information and integrate the semantic information of the context to obtain the context feature representation Z for the question.

[0134] like Figure 3A As shown, using the contextual reasoning module 330, according to the feature representation of each word Get the context feature representation Z for the problem.

[0135] For example, the weighted feature representation of each paragraph word is obtained Input into a bidirectional LSTM recurrent network, further encode and fuse the semantic information of the encoded question feature U and the encoded context feature H, and output a context feature representation for the question

[0136] In the embodiment of the present disclosure, for the multi-layer attention module, according to the similarity matrix E of the j-1th layer attention j -1 And the similarity matrix S j-1 , calculate the similarity matrix E of the j-th layer attention module j And the similarity matrix S j , j is greater than or equal to 1 and less than or equal to J, where J is the total number of layers of the attention module.

[0137] In operation S204, the information inference layer is used to obtain the encoded context paragraph feature M for the context feature Z of the question word.

[0138] In the embodiment of the present disclosure, the information inference layer is mainly used to infer the context feature Z of the question word to capture the semantic information of the question and context sequence. In this layer, a two-layer bidirectional Transformer network structure is used as an encoder for the semantic information between the context and the question. The context feature Z for the question word is input into the encoder, and the question information is further integrated into the context paragraph to obtain the encoded context paragraph feature.

[0139] In operation S205, the encoded context paragraph feature M is processed using the output layer to obtain and output the position of the question word in the context sequence.

[0140] In the embodiment of the present disclosure, the main function of the output layer is to predict the true answer to the question in the context. The output layer is mainly used to predict the specific starting position and ending position of the answer to the question in the context sequence. The pointer network is used as the answer prediction network structure.

[0141] For example, use the attention pooling operation to encode the feature U encoded as the title to generate the initial hidden state of the pointer network Afterwards, the encoded context paragraph feature M output by the information inference layer is input into the pointer network to calculate the starting position p of the answer in the context paragraph 1 , the specific calculation is shown in formula (10):

[0142]

[0143] After obtaining the probability distribution of the starting position of the answer, the initial state v of the answer end position prediction module is generated based on the probability distribution of the starting position and the output vector of the LSTM network. 2 =∑p 1 V, where is the hidden state of the LSTM network output in the answer starting position prediction module. Finally, the answer ending position probability p is calculated based on the probability distribution vector of the starting position. 2 , as shown in formula (11):

[0144]

[0145] Among them, w1, w2, is a trainable weight parameter.

[0146] The probability distribution of the starting and ending positions of the answer output by the output layer is used to establish the loss function of the model using maximum likelihood estimation. Its objective function is shown in formula (12):

[0147]

[0148] Among them, θ is all the trainable parameters in the model, are respectively the actual starting and ending positions of the answer in the context paragraph in the i-th training instance.

[0149] In an embodiment of the present disclosure, the method of analyzing text information using an attention model also includes using the position of a question word in a context sequence to determine at least one question corresponding to the question word so as to provide an answer to the at least one question.

[0150] In an embodiment of the present disclosure, during the communication process between customer service and the customer, or during the interaction between the customer and the automatic telephone customer service, the audio information in the conversation or interaction process is converted into a text file. The above text file is then analyzed by the method of analyzing text information using an attention model in an embodiment of the present disclosure. Then, according to the question raised by the customer, context information is extracted based on the question, and finally the content related to the question raised by the customer is displayed to the customer. The method of analyzing text information using the attention model provided in the embodiment of the present disclosure can extract more accurate context information for the question.

[0151] Figure 4 The structural diagram of the bidirectional Transformer module in the attention model according to an embodiment of the present disclosure is schematically shown.

[0152] like Figure 4 As shown, the bidirectional Transformer module 400 includes a self-attention module 410, a feedforward neural network 420, and an output module 430.

[0153] Among them, the self-attention module 410 is used to combine with the convolutional neural network to realize context feature extraction. The feedforward neural network 420 is used to perform feedback processing on the input question vector Q and context vector P. The output module 430 is used to output the data in the bidirectional Transformer module.

[0154] Figure 5 A block diagram of an apparatus for analyzing text information using an attention model according to an embodiment of the present disclosure is schematically shown.

[0155] like Figure 5 As shown, the device 500 for analyzing text information using an attention model includes a first module 510 , a second module 520 , a third module 530 , a fourth module 540 and a fifth module 550 .

[0156] The first module 510 is configured to use the input layer to convert the question words and context sequences included in the input text information into a question vector Q and a context vector P respectively.

[0157] The second module 520 is configured to use the context coding layer to encode and extract features from the question vector and the context vector respectively, to obtain encoded question features and encoded context features.

[0158] The third module 530 is configured to use the information interaction layer to fuse the encoded question feature U and the encoded context feature H to obtain the context feature Z for the question word.

[0159] The fourth module 540 is configured to obtain the encoded context paragraph feature M for the context feature Z using the information inference layer.

[0160] The fifth module 550 is configured to process the encoded context paragraph features using the output layer to obtain and output the position of the question word in the context sequence.

[0161] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0162] For example, any multiple of the first module 510, the second module 520, the third module 530, the fourth module 540, and the fifth module 550 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first module 510, the second module 520, the third module 530, the fourth module 540, and the fifth module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the first module 510, the second module 520, the third module 530, the fourth module 540 and the fifth module 550 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0163] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0164] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0165] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0166] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0167] The present disclosure also provides a computer-readable storage medium, which may be included in the device / electronic device described in the above embodiment; or may exist independently without being assembled into the device / electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0168] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0169] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the method of analyzing text information using an attention model provided by the embodiment of the present disclosure.

[0170] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0171] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0172] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0173] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods, devices and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0174] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0175] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for analyzing text information using an attention model, wherein the attention model includes an input layer, a context encoding layer, an information interaction layer, an information reasoning layer, and an output layer, wherein: The information interaction layer includes stacked multi-layer attention modules, the multi-layer attention modules are connected in series, each layer of attention modules includes a question-context alignment module, a context self-alignment module and a context reasoning module, the question-context alignment module, the context self-alignment module and the context reasoning module are connected in series, and the method includes: Using the input layer, the question words and context sequences included in the input text information are converted into a question vector Q and a context vector P respectively; and Use the context coding layer to encode and extract features from the question vector Q and the context vector P respectively, to obtain a coded question feature U and a coded context feature H; Using the information interaction layer to fuse the encoded question feature U and the encoded context feature H, to obtain a context feature Z for the question word; Obtaining an encoded context paragraph feature M for the context feature Z using the information inference layer; and Using the output layer to process the encoded context paragraph feature M to obtain and output the position of the question word in the context sequence, The step of fusing the coded question feature U and the coded context feature H using the information interaction layer to obtain the context feature Z for the question word includes: Using the question and context alignment module, obtaining a context paragraph representation V for the question word according to the encoded question feature U and the encoded context feature H; Using the context self-alignment module, the feature representation of each word in the context sequence is obtained according to the encoded context feature H and the context paragraph representation V. ;as well as Using the contextual reasoning module, according to the feature representation of each word , and obtain the context feature representation Z for the question word.

2. The method according to claim 1, wherein: The context encoding layer includes multiple bidirectional Transformer modules, each of which includes a self-attention module, a feedforward neural network and an output module.

3. The method according to claim 2, wherein: The step of using the question and context alignment module to obtain a context paragraph representation V for a question word according to the encoded question feature U and the encoded context feature H comprises: Calculating a first similarity matrix E between the coding problem feature U and the coding context feature H; Calculate the attention distribution of each word in the context sequence about the question word according to the similarity matrix E ;as well as According to the attention distribution Calculate the context paragraph representation V for the question word.

4. The method according to claim 3, wherein: The calculating the context paragraph representation V for the question word according to the attention distribution includes: According to the encoding problem feature U and the attention distribution , calculate the context paragraph representation matrix of each word in the context sequence with respect to the question word ; By representing the context paragraph matrix The elements in are weighted and summed to calculate the context paragraph representation V for the question word.

5. The method according to claim 4, wherein: The feature representation of each word in the context sequence is calculated based on the encoded context feature H and the context paragraph representation V. include: Calculate the similarity matrix S between each word in the context sequence; Calculate the attention distribution of each word in the context sequence with respect to the context sequence according to the similarity matrix S ;as well as According to the attention distribution Calculate the feature representation of each word in the context sequence .

6. The method according to claim 5, wherein: For the multi-layer attention module, according to the similarity matrix E of the j-1th layer attention j-1 And the similarity matrix S j-1 , calculate the similarity matrix E of the j-th layer attention module j And the similarity matrix S j , j is greater than or equal to 1 and less than or equal to J, where J is the total number of layers of the attention module.

7. The method according to any one of claims 1 to 6, further comprising: At least one question corresponding to the question word is determined using the position of the question word in the context sequence so as to provide an answer to the at least one question.

8. A device for analyzing text information using an attention model, wherein the attention model comprises an input layer, a context encoding layer, an information interaction layer, an information reasoning layer, and an output layer, wherein: The information interaction layer includes stacked multi-layer attention modules, which are connected in series. Each layer of attention modules includes a question-context alignment module, a context self-alignment module, and a context reasoning module. The question-context alignment module, the context self-alignment module, and the context reasoning module are connected in series. The device includes: A first module is used to convert the question words and context sequences included in the input text information into a question vector Q and a context vector P respectively using an input layer; and The second module is used to use the context coding layer to encode and extract features from the question vector and the context vector respectively, so as to obtain a coded question feature and a coded context feature; The third module is used to fuse the encoded question feature U and the encoded context feature H using the information interaction layer to obtain a context feature Z for the question word; A fourth module is used to obtain an encoded context paragraph feature M for the context feature Z using the information inference layer; and A fifth module is used to process the encoded context paragraph features using the output layer to obtain and output the position of the question word in the context sequence, The step of fusing the coded question feature U and the coded context feature H using the information interaction layer to obtain the context feature Z for the question word includes: Using the question and context alignment module, obtaining a context paragraph representation V for the question word according to the encoded question feature U and the encoded context feature H; Using the context self-alignment module, the feature representation of each word in the context sequence is obtained according to the encoded context feature H and the context paragraph representation V. ;as well as Using the contextual reasoning module, according to the feature representation of each word , and obtain the context feature representation Z for the question word.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 7.

11. A computer program product, wherein: The product stores a computer program, which can implement the method according to any one of claims 1 to 7 when executed.

Images