Text annotation method and neural network model construction method

Abstract

The present disclosure provides a text annotation method and a method for constructing a neural network model. The text annotation method includes: obtaining a text to be annotated, the text to be annotated includes a character sequence composed of multiple characters; converting the character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence; determining a second input vector that can represent context information of the character sequence according to a predetermined dictionary; using the first input vector and the second input vector as inputs of a bidirectional long short-term memory network model, outputting a feature vector of the character sequence; and determining the annotation information for each character in the character sequence using a conditional random field model according to the feature vector.

Description

Technical Field

[0001] The present disclosure relates to the field of electronic technology, and more specifically, to a text annotation method and a method for constructing a neural network model. Background Art

[0002] With the development of the Internet of Things (IoT), the way people use machines is becoming increasingly intelligent. Technologies such as smart homes and smart vehicles are gaining increasing attention. NLP (Neuro-Linguistic Programming) technology is a key enabler for the development of these intelligent technologies. Most NLP problems can be converted into text annotation problems. Deep learning and machine learning have become mainstream technologies for text annotation because they offer higher accuracy, greater flexibility, and greater applicability than traditional methods.

[0003] Although convolutional neural network models can leverage the relevance of text context when used for text annotation in related technologies, their generalization ability is poor when there are many annotation categories, the categories are difficult to distinguish, and the amount of data is small. This can affect the accuracy of text annotation to a certain extent. Summary of the Invention

[0004] In view of this, the present disclosure provides a text annotation method capable of improving the accuracy of text annotation and a method for constructing a neural network model capable of performing text annotation.

[0005] One aspect of the present disclosure provides a text annotation method, which includes: obtaining a text to be annotated, the text to be annotated including a character sequence consisting of multiple characters; converting the character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence; determining a second input vector that can represent context information of the character sequence according to a predetermined dictionary; using the first input vector and the second input vector as inputs of a bidirectional long short-term memory network model to output a feature vector of the character sequence; and determining, based on the feature vector, using a conditional random field model to determine annotation information for each character in the character sequence.

[0006] According to an embodiment of the present disclosure, a predetermined dictionary includes multiple phrases, each of which is for a plurality of categories, and the annotation information for each character can represent one of the plurality of categories. The above-mentioned determination of the second input vector capable of representing the context information of the text to be annotated based on the predetermined dictionary includes: determining a word set for each character based on each character in the character sequence and the characters adjacent to each character in the character sequence; determining a second word vector for each character based on whether each word included in the word set for each character belongs to a plurality of phrases; and merging a plurality of second word vectors for a plurality of characters based on the distribution order of the plurality of characters in the character sequence to obtain a second input vector for the character sequence.

[0007] According to an embodiment of the present disclosure, determining a word set for each character includes: obtaining n first characters and n second characters preceding each character in a character sequence; and determining at least one word representing contextual information for each character based on the n first characters, the n second characters, and each character, to obtain a word set for each character. Each of the at least one word includes at least two characters, and the at least two characters are adjacent in the character sequence. Wherein, n is a positive integer.

[0008] According to an embodiment of the present disclosure, each of the above-mentioned multiple phrases has a pre-assigned identification code; the above-mentioned determination of the second word vector for each character includes: determining a target word belonging to multiple phrases among the respective words included in the word set for each character; assigning a feature value to the target word according to the identification code of the phrase to which the target word belongs in the multiple phrases; assigning predetermined feature values ​​to the other words except the target word included in the word set for each character; and determining the second word vector for each character according to the feature values ​​assigned to the respective words included in the word set for each character.

[0009] According to an embodiment of the present disclosure, the above-mentioned determination of the labeling information for each character in the character sequence based on the feature vector using the conditional random field model includes: determining the attention vector of the feature vector using the self-attention mechanism; determining the vector obtained by multiplying the feature vector by the dot product of the attention vector as the optimized feature vector; and determining the labeling information for each character in the character sequence based on the optimized feature vector using the conditional random field model.

[0010] According to an embodiment of the present disclosure, using the first input vector and the second input vector as inputs of a bidirectional long short-term memory network model and outputting a feature vector of a character sequence includes: using the first input vector as input of a first bidirectional long short-term memory network model and outputting a first feature vector; using the second input vector as input of a second bidirectional long short-term memory network model and outputting a second feature vector; and merging the first feature vector and the second feature vector to obtain a feature vector of the character sequence.

[0011] According to an embodiment of the present disclosure, the above-mentioned conversion of the character sequence according to a predetermined conversion rule to obtain the first input vector for the character sequence includes: converting each character in the character sequence according to the predetermined conversion rule to determine the first word vector for each character; merging the first word vectors for multiple characters according to the distribution order of multiple characters in the character sequence to obtain the first input vector for the character sequence.

[0012] Another aspect of the present disclosure provides a method for constructing a neural network model, which includes: constructing an embedding layer, which is used to convert a character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence; constructing a preprocessing layer, which is used to determine a second input vector that can represent context information of the character sequence according to a predetermined dictionary; constructing a processing layer, which is used to use the first input vector and the second input vector as inputs of a bidirectional long short-term memory network model to output a feature vector of the character sequence; and constructing an output layer, which is used to determine the labeling information for each character in the character sequence based on the feature vector using a conditional random field model.

[0013] According to an embodiment of the present disclosure, the method for constructing the neural network model further includes: constructing an optimization layer, located between the processing layer and the output layer, comprising a first fully connected layer, an activation layer, and a logistic regression layer; the optimization layer is configured to determine the attention vector of the feature vector using a self-attention mechanism; the output layer comprises a computation layer, a second fully connected layer, and a conditional random field decoding layer. The computation layer is configured to determine the vector obtained by multiplying the attention vector by the feature vector as the optimized feature vector; the second fully connected layer and the conditional random field decoding layer are configured to determine the labeling information for each character in the character sequence using the conditional random field model based on the optimized feature vector.

[0014] According to an embodiment of the present disclosure, the processing layer includes a first bidirectional long short-term memory network layer and a second bidirectional long short-term memory network layer. The first bidirectional long short-term memory network layer is configured to take a first input vector as input and output a first feature vector; the second bidirectional long short-term memory network layer is configured to take a second input vector as input and output a second feature vector. The feature vector of the character sequence is obtained by combining the first feature vector and the second feature vector.

[0015] On the other hand, the present disclosure provides a computer system comprising: one or more processors; and a storage device for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned text annotation method and / or neural network model construction method.

[0016] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to execute the text annotation method and / or the neural network model construction method as described above.

[0017] Another aspect of the present disclosure provides a computer program, which includes computer-executable instructions. When the instructions are executed, the computer program is used to implement the method described above.

[0018] According to the embodiments of the present disclosure, the technical problem of low accuracy in text annotation in related technologies can be at least partially resolved. By obtaining a second input vector representing contextual information of a character sequence based on a predetermined dictionary during text annotation, the disclosed embodiments can effectively extract word features from a character sequence. Consequently, when using a bidirectional long short-term memory network model and a conditional random field model to obtain annotation information, more comprehensive feature information can be input, thereby improving the accuracy of the obtained annotation information and, in turn, the accuracy of the annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1 Schematically illustrates the application scenarios of the text annotation method and the neural network construction method according to the embodiments of the present disclosure;

[0021] Figure 2 The flowchart of the text annotation method according to the embodiment of the present disclosure is schematically shown;

[0022] Figure 3 Schematically shows a flow chart for determining a second input vector capable of representing context information of a text to be annotated according to an embodiment of the present disclosure;

[0023] Figure 4 A flowchart of outputting a feature vector of a character sequence according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 5 A flowchart of determining labeling information for each character in a character sequence using a conditional random field model according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 6 A schematic diagram of the structure of a neural network model constructed according to the method for constructing a neural network model according to an embodiment of the present disclosure is shown;

[0026] Figure 7A schematic diagram schematically illustrates the structure of a neural network model constructed according to a method for constructing a neural network model according to another embodiment of the present disclosure; and

[0027] Figure 8 A block diagram of a computer system suitable for implementing a text annotation method and / or a neural network model construction method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] An embodiment of the present disclosure provides a text annotation method. First, a text to be annotated is obtained, where the text to be annotated includes a character sequence consisting of multiple characters. Then, the character sequence is converted according to a predetermined conversion rule to obtain a first input vector for the character sequence; and according to a predetermined dictionary, a second input vector capable of representing contextual information of the character sequence is determined. Subsequently, the first input vector and the second input vector are used as inputs of a bidirectional long short-term memory network model, and a feature vector of the character sequence is output. Finally, based on the feature vector, a conditional random field model is used to determine the annotation information for each character in the character sequence.

[0033] Figure 1 The following schematically illustrates the application scenarios of the text annotation method and the neural network construction method according to the embodiments of the present disclosure. Figure 1 The examples shown are merely examples of application scenarios in which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101 , 102 , and 103 .

[0035] For example, various communication client applications may be installed on the terminal devices 101 , 102 , and 103 , such as smart home control applications, web browser applications, search applications, etc. (only as examples).

[0036] According to an embodiment of the present disclosure, the terminal devices 101, 102, and 103 can be used, for example, to collect user voice information and convert the voice information into text. The user's voice information can be recognized by performing sequence annotation on the text to obtain a recognition result, and the user's voice information can be responded to in response to the recognition result.

[0037] According to an embodiment of the present disclosure, the terminal devices 101, 102, and 103 may also construct a neural network model in response to user interaction operations, and train the model to perform sequence tagging on text based on the trained neural network model.

[0038] According to the embodiments of the present disclosure, Figure 1 As shown, the application scenario 100 may further include a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and wireless communication links.

[0039] Among them, users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Server 105 can be a server that provides various services, such as a background management server that supports text annotation performed by terminal devices 101, 102, 103 (only as an example).

[0040] In one embodiment, the backend management server may feedback the model required for text annotation to the terminal device in response to the received acquisition request, etc. The model may be, for example, a neural network model, which may be pre-built and trained by the server 105.

[0041] In one embodiment, the background management server may, for example, respond to receiving the text to be annotated sent by the terminal devices 101 , 102 , 103 , and annotate the text to be annotated according to the neural network model.

[0042] It should be noted that the text annotation method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, 103 or the server 105. The neural network model construction method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, 103 or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0044] The following will be combined Figures 2 to 5 The text annotation method of the embodiment of the present disclosure is described in detail.

[0045] Figure 2 The flowchart of the text annotation method according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the text annotation method of this embodiment may include operations S210 to S250.

[0047] In operation S210 , a text to be annotated is obtained, where the text to be annotated includes a character sequence consisting of a plurality of characters.

[0048] The text to be annotated may be, for example, text information converted from voice information input by a user. The text to be annotated includes a plurality of characters, which form a character sequence.

[0049] In operation S220 , the character sequence is converted according to a predetermined conversion rule to obtain a first input vector for the character sequence.

[0050] According to an embodiment of the present disclosure, operation S220 may, for example, first convert each character in the character sequence according to a predetermined conversion rule to determine a first word vector for each character. Then, based on the distribution order of the multiple characters in the character sequence, the first word vectors for the multiple characters are merged to obtain a first input vector for the character sequence.

[0051] Among them, according to the predetermined conversion rule, the first word vector for each character is determined to be converted into a corresponding digital id according to the corpus, thereby obtaining a digital sequence composed of multiple digital ids. For example, if a character in the character sequence is the third character in the corpus, the digital id corresponding to the character can be 3. Then the digital id is used as the input of the embedding layer, and the digital id is upgraded to an m-dimensional vector via the embedding layer, and the m-dimensional vector is used as the first word vector for each character. The value of m can be, for example, the number of characters included in the corpus. After obtaining the first word vector for each character, the first word vector for the first character in the character sequence is used as the first row of data of the first input vector, the first word vector for the second character is used as the second row of data of the first input vector, and so on to obtain the first input vector for the character sequence. The number of rows of the first input vector is the number of characters included in the character sequence.

[0052] According to an embodiment of the present disclosure, the operation S220 may also convert all characters in the character sequence into corresponding digital IDs according to the expected library to obtain a data ID sequence. Then the digital ID sequence is used as Figures 6 and 7 The input of the embedding layer in the described neural network model is processed by the embedding layer to obtain the first input vector.

[0053] In operation S230 , a second input vector capable of representing context information of the character sequence is determined according to a predetermined dictionary.

[0054] According to an embodiment of the present disclosure, the predetermined dictionary can be obtained by, for example, extracting words from a large amount of corpus and grouping the extracted words. By grouping a large amount of words, the predetermined dictionary can, for example, include multiple phrases. In one embodiment, the large amount of words can be grouped according to predetermined annotation information. For example, if the annotation information can be used to represent the category of an entity, and there are n categories of entity categories, then the predetermined dictionary can include n phrases. The n phrases in the predetermined dictionary are respectively for the n categories, that is, the n phrases correspond one-to-one to the n categories, and the words included in a certain phrase in the n phrases all belong to the category corresponding to the certain phrase. Therefore, when annotating each character in the text to be annotated, the annotation information used can represent one of the n categories. Considering that there may be characters in the text to be annotated that cannot be used to represent an entity, the n categories can, for example, include categories that do not belong to entities. Accordingly, the words included in a certain phrase in the n phrases in the predetermined dictionary cannot be used to represent an entity.

[0055] According to an embodiment of the present disclosure, operation S230 may, for example, include: first performing word segmentation processing on the character sequence to obtain all words that can be formed by combining all characters in the character sequence, each word can include at least two characters; then determining the distribution of all words in multiple phrases in a predetermined dictionary; and finally obtaining a second input vector based on the distribution. Since each word is formed by at least two characters, when multiple phrases include a word from all the words, the category to which the word belongs can be known. Therefore, the second input vector obtained based on the distribution of all the words can reflect the contextual information of the character sequence to a certain extent.

[0056] In one embodiment, the operation S230 may, for example, directly use the character sequence as Figures 6 and 7 The input of the preprocessing layer in the step 2 is processed by the preprocessing layer to obtain a second input vector. Operation S230 can be performed by, for example, Figure 3 The process described is implemented and will not be described in detail here.

[0057] In operation S240, the first input vector and the second input vector are used as inputs of a bidirectional long short-term memory network model, and a feature vector of the character sequence is obtained as output.

[0058] According to an embodiment of the present disclosure, operation S240 may include, for example, sequentially inputting the first input vector and the second input vector into a pre-trained bidirectional long short-term memory (Bi-LSTM) network model to sequentially obtain a feature vector extracted from the first input vector and a feature vector extracted from the second input vector. The two feature vectors obtained in sequence are then concatenated to obtain a feature vector for the character sequence.

[0059] According to an embodiment of the present disclosure, operation S240 may also first merge the first input vector and the second input vector to form an input vector, and then input the input vector into a bidirectional long short-term memory network model to output a feature vector of the character sequence.

[0060] In one embodiment, the operation S240 may be performed by, for example, Figure 7 The first bidirectional long short-term memory network layer and the second bidirectional long short-term memory network layer in the described neural network model are then Figure 4 The process described is implemented and will not be described in detail here.

[0061] In operation S250 , labeling information for each character in the character sequence is determined using a conditional random field model according to the feature vector.

[0062] According to an embodiment of the present disclosure, a conditional random fields model, for example, can be a processing model adopted by a conditional random fields decoding layer in a subsequent neural network model. Among them, in order to facilitate determining annotation information through this conditional random fields model, after obtaining the feature vector, the vector represented by the feature vector can also be abstracted to map the feature vector to the space composed of annotation information. Then, the vector obtained after mapping the feature vector is used as the input variable of the CRF model, and an annotation sequence is output by the CRF model. The annotation sequence includes the id serial numbers of the annotation information for each character. Finally, according to the id serial number and the annotation information-id serial number mapping table, the annotation information for each character can be obtained. Among them, the annotation information-id serial number mapping table can be constructed in advance, for example. Figure 7 In one embodiment, the annotation method described in the foregoing operations S210 to S250 can be used to annotate the entity categories in the text to be annotated, for example. Among them, the annotation information can be in the BI annotation method or the BME method, for example. When using the BI annotation method, if the text to be annotated includes the entity word "movie", the annotation information for the character "电" in the entity word "movie" is "B-VIDEO_TYPE", which is used to represent that the character "电" is the starting character of the media category in the entity category. The annotation information for the character "影" in the entity word "movie" is "I-VIDEO_TYPE", which is used to represent that the character "影" is the subsequent character of the media category in the entity category. For words in the text to be annotated that cannot be used to represent entities, such as words like "的" and "了", the corresponding annotation information can be "O", for example.

[0063] In one embodiment, operation S250 can be implemented, for example, by

[0064] the output layer in the neural network described by Figure 6 and Figure 7 which will not be elaborated here.

[0065] In summary, in the text annotation method of the embodiment of the present disclosure, when using a bidirectional long short-term memory network model and a conditional random fields model to determine the annotation information of characters, by incorporating the second input vector that can represent context information obtained according to a predetermined dictionary, the information of the annotation category to which a known word belongs can also be introduced into the neural network, thereby effectively improving the accuracy of the annotation information.

[0066] Figure 3 FIG. schematically shows a flowchart of determining a second input vector that can represent the context information of the text to be annotated according to an embodiment of the present disclosure.

[0067] As Figure 3As shown, the second input vector that can represent the context information of the text to be annotated may include operations S331 to S333, for example.

[0068] In operation S331, according to each character in the character sequence and the characters adjacent to each character in the character sequence, a word set for each character is determined.

[0069] According to an embodiment of the present disclosure, in operation S331, for each character, n first characters located before the each character and n second characters located after the each character in the character sequence may be determined first, where n is a positive integer. Then, at least one word representing the context information of each character is determined according to the n first characters, the n second characters and each character, so as to obtain a word set for each character. Each word in the at least one word includes at least two characters, and the at least two characters are adjacent in position in the character sequence.

[0070] In one embodiment, n may be taken as 2, for example, and the text to be annotated may be "Open the video about Teddy". Correspondingly, the character sequence is [Open the video about Teddy]. For the character "Tai" in the character sequence, the obtained first characters include "Guan" and "You", and the obtained second characters include "Di" and "De". According to the first characters and the second characters, the word set for the character "Tai" includes the following words: "Guan Tai", "You Guan Tai", "Teddy", "Teddy De".

[0071] According to an embodiment of the present disclosure, when determining the word set for the first character in the character sequence, considering that there is no character before the first character, the obtained first characters are an empty set. Correspondingly, for the word set of the first character, it only includes the words composed of the first character and the second characters.

[0072] In operation S332, according to whether each word included in the word set for each character belongs to multiple word groups, a second word vector for each character is determined.

[0073] According to an embodiment of the present disclosure, each element in the second word vector may be used to represent whether a certain word in the word set for each character belongs to multiple word groups in a predetermined dictionary. In order to further enable the element to represent which word group a certain word belongs to among the multiple word groups, the embodiment may also pre-assign identification codes to each word group in the multiple word groups. The pre-assigned identification code starts from 1 and increases by 1 at an interval.

[0074] The operation S332 can, for example, first select the words belonging to a predetermined dictionary from the word sets for each character. Specifically, determine the target words belonging to multiple word groups among the words included in the word set for each character. Then, according to the identification codes of the word groups to which the target words belong among the multiple word groups, assign feature values to the target words. For example, if a certain target word belongs to the word group with the identification code 2 in the predetermined dictionary, the feature value assigned to this certain target word is 2. Among them, considering that some words in the word set for each character do not belong to the predetermined dictionary, therefore, a predetermined feature value can be assigned to the other words except the target words included in the word set for each character. The predetermined feature value can be, for example, 0, which is used to indicate that the other words do not belong to the predetermined dictionary. Finally, according to the feature values assigned to the words included in the word set for each character, determine the second word vector for each character.

[0075] In one embodiment, if the entity categories include the actor category, the media category, the writer category, and the animal category. Correspondingly, there are five word groups, and the words included in four of the five word groups respectively correspond to the aforementioned four entity categories, and the words included in the other word group cannot represent an entity. In this case, the identification codes assigned to the four word groups corresponding to the four entity categories can be, for example, 1 (the identification code of the word group corresponding to the actor category), 2 (the identification code of the word group corresponding to the media category), 3 (the identification code of the word group corresponding to the writer category), 4 (the identification code of the word group corresponding to the plant category). Among them, considering that the words that do not belong to an entity will not have a substantial effect on the accuracy of the annotation information, the word group whose included words cannot represent an entity can not be assigned an identification code. Therefore, when the word set for the character "泰" includes the following words: "有关泰", "关泰", "泰迪", "泰迪的", the determined target word only includes, for example, "泰迪". Since "泰迪" is a word representing an animal, the assigned feature value is 4, and the feature values assigned to the other three words "关泰", "有关泰", "泰迪的" are 0. The finally obtained second word vector for the character "泰" is [0, 0, 4, 0].

[0076] According to an embodiment of the present disclosure, in the word set for each character, the distribution order of each word can be arranged, for example, according to the order in which the characters forming each word appear in the character sequence. The words composed of the characters located in the relatively front position in the character sequence are distributed in the front position in the word set. Correspondingly, the distribution positions of the elements in the second word vector are consistent with the distribution positions of the words in the word set.

[0077] In operation S333, according to the distribution order of the multiple characters in the character sequence, merge the multiple second word vectors for the multiple characters to obtain the second input vector for the character sequence.

[0078] According to an embodiment of the present disclosure, operation S333 may include: taking the second word vector for the first character in the character sequence as an element of the first row, taking the second word vector for the second character in the character sequence as an element of the second row, and so on, merging multiple second word vectors for multiple characters to obtain a second input vector for the character sequence.

[0079] In summary, the method of obtaining the second input vector according to a predetermined dictionary in the embodiment of the present disclosure can effectively integrate known word knowledge into the text annotation method, thereby providing more comprehensive context information for the text annotation method, facilitating the effective distinction of different categories of annotations in the annotated text, and can improve the annotation accuracy to a certain extent and improve the universality of the annotation method.

[0080] According to an embodiment of the present disclosure, considering that the first input vector and the second input vector are obtained in different ways, in order to improve the accuracy of feature extraction, two different bidirectional long short-term memory network models can be pre-trained to perform feature extraction based on the two input vectors.

[0081] Figure 4 The flowchart of outputting a feature vector of a character sequence according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 4 As shown, the operation of outputting the feature vector of the character sequence may include, for example, operations S441 to S443.

[0083] In operation S441, the first input vector is used as the input of a first bidirectional long short-term memory network model, and a first feature vector is output.

[0084] According to an embodiment of the present disclosure, the first long short-term memory network model can be, for example, trained by using a large number of first input vectors obtained by converting a large number of character sequences of annotated texts through the method described in operation S220 as sample data. The initial parameters of the first long short-term memory network model can be set according to actual needs, and the present disclosure does not limit this. In one embodiment, the operation S441 can, for example, use the first input vector as Figure 7 The input of the first bidirectional long short-term memory network layer in is processed by the first bidirectional long short-term memory network layer and outputs a first eigenvector.

[0085] In operation S442, the second input vector is used as input of a second bidirectional long short-term memory network model, and a second feature vector is output.

[0086] According to an embodiment of the present disclosure, the first long short-term memory network model can be, for example, trained using a large number of second input vectors obtained by converting a large number of character sequences of annotated texts using the method described in operation S230 as sample data. The initial parameters of the second long short-term memory network model can be set according to actual needs, and the present disclosure does not limit this. The initial parameters of the second long short-term memory network model can be the same as or different from the initial parameters of the first long short-term memory network model, and the present disclosure does not limit this.

[0087] According to an embodiment of the present disclosure, to facilitate the subsequent merging of the two eigenvectors, the number of cores in the last layer of the first LSTM network model can be equal to the number of cores in the last layer of the second LSTM network model, thereby making the number of columns of the first eigenvector and the second eigenvector equal.

[0088] In one embodiment, the operation S442 may, for example, take the second input vector as Figure 7 The input of the second bidirectional long short-term memory network layer in is processed by the second bidirectional long short-term memory network layer and outputs a second eigenvector.

[0089] In operation S443, the first feature vector and the second feature vector are combined to obtain a feature vector of the character sequence. Operation S443 may, for example, concatenate the second feature vector to the next row of the last row of the first feature vector to obtain the feature vector of the character sequence.

[0090] According to an embodiment of the present disclosure, different features among the multiple features included in the obtained feature vector have different importance for determining the annotation information. In order to highlight the difference in importance, weights can be assigned to the elements in the feature vector so that when the conditional random field model is used to determine the annotation information of each character, more attention is paid to the contribution of elements with large weights, thereby making the final annotation information more accurate. The weights assigned to the elements can be determined, for example, by a self-attention mechanism.

[0091] Figure 5 The flowchart of determining the labeling information for each character in a character sequence using a conditional random field model according to an embodiment of the present disclosure is schematically shown.

[0092] like Figure 5 As shown, the operation of determining the tag information for each character in the character sequence using the conditional random field model may include, for example, operations S551 to S553.

[0093] In operation S551, a self-attention mechanism is used to determine an attention vector of a feature vector.

[0094] According to an embodiment of the present disclosure, the operation S551 may, for example, take the feature vector as Figure 7 The input of the first fully connected layer in the attention vector is used as the input of the activation layer, and the output of the activation layer is used as the input of the logistic regression layer. After being processed by the first fully connected layer, the activation layer and the logistic regression layer, an attention vector is obtained. The activation function used by the activation layer may include, for example, the tanh activation function, the sigmoid activation function, etc. The logistic regression layer can be used to normalize the value of each element in the attention vector to the interval [0, 1]. The value of an element in the obtained attention vector is used to represent the weight of the element in the feature vector that is at the same position as the element. The dimension of the attention vector is equal to the dimension of the feature vector.

[0095] In operation S552, a vector obtained by multiplying the attention vector by the feature vector is determined as the optimized feature vector. According to an embodiment of the present disclosure, operation S552 can be performed, for example, by Figure 7 The output layer includes the calculation layer to perform.

[0096] In operation S553 , labeling information for each character in the character sequence is determined using a conditional random field model according to the optimized feature vector.

[0097] After obtaining the optimized feature vector, it is first used as the input to the second fully connected layer in the figure to abstract it. The abstracted vector is then used as the input variable of the conditional random field decoding layer. After processing by the conditional random field decoding layer, a label sequence is obtained. This label sequence includes the ID number of the label information for each character. Finally, the label information for each character is obtained based on the ID number and the label information-ID number mapping table.

[0098] According to an embodiment of the present disclosure, in order to improve the annotation efficiency, the present disclosure may also pre-construct a neural network model for performing text annotation, for example. Figures 2 to 5 The text annotation method described in can be performed by using this neural network model.

[0099] Figure 6 The schematic diagram shows the structure of the neural network model constructed according to the method for constructing the neural network model of an embodiment of the present disclosure.

[0100] According to an embodiment of the present disclosure, the construction method may include, for example, constructing the following layers: an embedding layer, a preprocessing layer, a processing layer, and an output layer. The constructed neural network model 600 is as follows: Figure 6 shown.

[0101] The embedding layer 610 is used to convert the character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence. The embedding layer 610 can, for example, use the digital sequence converted from the character sequence of the text to be annotated obtained in operation S210 as the input of the embedding layer 610, and output the first input vector after processing. The embedding layer 610 can be used to perform Figure 2 Operation S220 described above will not be repeated here.

[0102] The pre-processing layer 620 is used to determine a second input vector that can represent the context information of the character sequence according to a predetermined dictionary. The pre-processing layer 620 can be used to perform Figure 2 The operation S230 described above will not be repeated here.

[0103] The processing layer 630 is used to take the first input vector and the second input vector as the input of the bidirectional long short-term memory network model and output the feature vector of the character sequence. The processing layer 630 can be used to perform Figure 2 Operation S240 described above will not be described again here.

[0104] The output layer 640 is used to determine the label information for each character in the character sequence using the conditional random field model based on the feature vector. The output layer 640 can be used to perform Figure 2 Operation S250 described above will not be described again in detail.

[0105] Figure 7 The schematic diagram shows the structure of a neural network model constructed according to the method for constructing a neural network model in another embodiment of the present disclosure.

[0106] According to an embodiment of the present disclosure, the construction method may further include, for example, constructing Figure 7 The optimized layer 750 shown in FIG. Figure 7 The neural network model 700 shown. The optimization layer 750 is located between the processing layer 630 and the output layer 640. Specifically, the optimization layer 750 may include, for example, a fully connected layer 751, an activation layer 752, and a logistic regression layer 753. The optimization layer 750 is used to determine the attention vector of the feature vector using a self-attention mechanism. The fully connected layer 751 is used to perform dimensionality conversion on the feature vector, and the activation layer 752 may use, for example, an activation function such as tanh or sigmoid to process the feature vector after dimensionality conversion to increase the nonlinearity of the neural network model, so as to enhance the expressive power of the neural network model, so that the processed vector can more accurately represent the importance of each element in the feature vector. The logistic regression layer can be used to normalize the value of each element in the processed vector to the interval [0, 1], thereby obtaining an attention vector.

[0107] like Figure 7 As shown, the output layer 640 may include, for example, a computing layer 641, a second fully connected layer 642, and a conditional random field decoding layer 643. The computing layer 641 is used to determine the vector obtained by multiplying the attention vector by the feature vector as the optimized feature vector. The second fully connected layer 642 and the conditional random field decoding layer 643 are used to determine the labeling information for each character in the character sequence using the conditional random field model based on the optimized feature vector. Among them, the computing layer 641 may be used to perform, for example Figure 5 The second fully connected layer 642 and the conditional random field decoding layer 643 are used to perform Figure 5 Operation S553 is described.

[0108] like Figure 7 As shown, in one embodiment, the processing layer 630 may include, for example, a first bidirectional long short-term memory network layer 631 and a second bidirectional long short-term memory network layer 632. The first bidirectional long short-term memory network layer 631 is used to take the first input vector as input and output a first feature vector. The second bidirectional long short-term memory network layer 632 is used to take the second input vector as input and output a second feature vector. The feature vector of the character sequence is obtained by combining the first feature vector and the second feature vector. The first bidirectional long short-term memory network layer 631 is used to perform Figure 4 In the described operation S441, the second bidirectional long short-term memory network layer 632 is used to perform Figure 4 Operation S442 is described.

[0109] In summary, the embodiment of the present disclosure can directly use the character sequence of the text to be annotated as the input of the neural network model by constructing a neural network model, and output the annotation information of each character in the character sequence. Compared with the model in the prior art, this model adds a second bidirectional long short-term memory network layer 632 in parallel with the first bidirectional long short-term memory network layer 631. Known word knowledge can be used when determining the annotation information, thereby improving the accuracy of the determined annotation information. Furthermore, by setting an optimization layer, the neural network model can realize the re-extraction of features in the feature vector, thereby highlighting the features that contribute greatly to the annotation information, and further improving the accuracy of the determined annotation information.

[0110] Figure 8 A block diagram of a computer system suitable for implementing a text annotation method and / or a neural network model construction method according to an embodiment of the present disclosure is schematically shown. Figure 8 The computer system shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0111] like Figure 8As shown, the computer system 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0112] Various programs and data required for the operation of the system 800 are stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. The processor 801 executes the programs in the ROM 802 and / or RAM 803 to perform various operations of the method flow according to the embodiment of the present disclosure. It should be noted that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also execute the programs stored in the one or more memories to perform various operations of the method flow according to the embodiment of the present disclosure.

[0113] According to an embodiment of the present disclosure, the system 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The system 800 may also include one or more of the following components connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 808 including a hard disk; and a communication section 809 including a network interface card such as a LAN card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 810 as needed, so that a computer program read therefrom can be installed into the storage section 808 as needed.

[0114] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0115] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0116] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0119] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A text annotation method, comprising: Acquire a text to be annotated, wherein the text to be annotated includes a character sequence consisting of a plurality of characters; Convert the character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence; Determining a second input vector capable of representing context information of the character sequence according to a predetermined dictionary, wherein the predetermined dictionary includes a plurality of phrases; The step of determining, according to a predetermined dictionary, a second input vector capable of representing the context information of the character sequence comprises: Determine a word set for each character according to each character in the character sequence and characters adjacent to each character in the character sequence; Determine a second word vector for each character according to whether each word included in the word set for each character belongs to the multiple word groups, wherein each element in the second word vector is used to represent whether a certain word in the word set for each character belongs to the multiple word groups in the predetermined dictionary, the length of the second word vector is equal to the number of words included in the word set for each character, and the vector value of the element in the second word vector represents the category of the word group to which a certain word in the word set for each character belongs; and Merging a plurality of second word vectors for the plurality of characters according to a distribution order of the plurality of characters in the character sequence to obtain a second input vector for the character sequence; Each of the plurality of word groups has a pre-assigned identification code; and determining the second word vector for each character comprises: Determine a target word belonging to the plurality of word groups among the words included in the word set for each character; assigning a feature value to the target word according to an identification code of the word group to which the target word belongs among the multiple word groups; assigning predetermined feature values ​​to other words, except the target word, included in the word set for each character; and Determine a second word vector for each character according to feature values ​​assigned to each word included in the word set for each character; The step of determining a word set for each character includes: Obtaining n first characters located before each character and n second characters located after each character in the character sequence; and According to the n first characters, the n second characters and each character, at least one word representing the context information of each character is determined to obtain a word set for each character, Wherein, each word in the at least one word includes at least two characters, and the positions of the at least two characters in the character sequence are adjacent; n is a positive integer; Taking the first input vector and the second input vector as inputs of a bidirectional long short-term memory network model, and outputting a feature vector of the character sequence, wherein the bidirectional long short-term memory network model comprises a first bidirectional long short-term memory network model taking the first input vector as input and a second bidirectional long short-term memory network model taking the second input vector as input, and the number of cores in the last layer of the first bidirectional long short-term memory network model is equal to the number of cores in the last layer of the second bidirectional long short-term memory network model; and A weight is assigned to the feature vector, and a conditional random field model is used to determine labeling information for each character in the character sequence.

2. The method according to claim 1, wherein: The multiple phrases are respectively for multiple categories, and the annotation information for each character can represent one of the multiple categories.

3. The method according to claim 1, wherein: Determining the labeling information for each character in the character sequence by using a conditional random field model according to the feature vector includes: Determine an attention vector of the feature vector using a self-attention mechanism; By multiplying the feature vector by the attention vector, a weight is assigned to the feature vector to obtain an optimized feature vector; and According to the optimized feature vector, a conditional random field model is used to determine the labeling information for each character in the character sequence.

4. The method according to claim 1, wherein: The first input vector and the second input vector are used as inputs of the bidirectional long short-term memory network model, and the feature vector of the character sequence is outputted, including: Using the first input vector as input of a first bidirectional long short-term memory network model, and outputting a first feature vector; Using the second input vector as input of a second bidirectional long short-term memory network model, and outputting a second feature vector; and The first feature vector and the second feature vector are combined to obtain a feature vector of the character sequence.

5. The method according to claim 1, wherein: The converting the character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence comprises: Convert each character in the character sequence according to a predetermined conversion rule, and determine a first word vector for each character; According to the distribution order of the multiple characters in the character sequence, first word vectors for the multiple characters are merged to obtain a first input vector for the character sequence.

6. A method for constructing a neural network model, comprising: Constructing an embedding layer, wherein the embedding layer is used to convert the character sequence according to a predetermined conversion rule to obtain a first input vector for the character sequence; Constructing a preprocessing layer, the preprocessing layer being used to determine a second input vector capable of representing context information of the character sequence according to a predetermined dictionary, wherein the predetermined dictionary includes a plurality of phrases; The step of determining, according to a predetermined dictionary, a second input vector capable of representing the context information of the character sequence comprises: Determine a word set for each character according to each character in the character sequence and characters adjacent to each character in the character sequence; Determine a second word vector for each character according to whether each word included in the word set for each character belongs to the multiple word groups, wherein each element in the second word vector is used to represent whether a certain word in the word set for each character belongs to the multiple word groups in the predetermined dictionary, the length of the second word vector is equal to the number of words included in the word set for each character, and the vector value of the element in the second word vector represents the category of the word group to which a certain word in the word set for each character belongs; and Merging a plurality of second word vectors for the plurality of characters according to a distribution order of the plurality of characters in the character sequence to obtain a second input vector for the character sequence; Each of the plurality of word groups has a pre-assigned identification code; and determining the second word vector for each character comprises: Determine a target word belonging to the plurality of word groups among the words included in the word set for each character; assigning a feature value to the target word according to an identification code of the word group to which the target word belongs among the multiple word groups; assigning predetermined feature values ​​to other words, except the target word, included in the word set for each character; and Determine a second word vector for each character according to feature values ​​assigned to each word included in the word set for each character; The step of determining a word set for each character includes: Obtaining n first characters located before each character and n second characters located after each character in the character sequence; and According to the n first characters, the n second characters and each character, at least one word representing the context information of each character is determined to obtain a word set for each character, Wherein, each word in the at least one word includes at least two characters, and the positions of the at least two characters in the character sequence are adjacent; n is a positive integer; Constructing a processing layer, the processing layer being used to use the first input vector and the second input vector as inputs of a bidirectional long short-term memory network model, and outputting a feature vector of the character sequence, wherein the bidirectional long short-term memory network model comprises a first bidirectional long short-term memory network model using the first input vector as input and a second bidirectional long short-term memory network model using the second input vector as input, and the number of cores in the last layer of the first bidirectional long short-term memory network model is equal to the number of cores in the last layer of the second bidirectional long short-term memory network model; and An output layer is constructed, wherein the output layer is used to assign weights to the feature vectors, and a conditional random field model is used to determine labeling information for each character in the character sequence.

7. The method according to claim 6, wherein: The method further includes: constructing an optimization layer, the optimization layer being located between the processing layer and the output layer, the optimization layer comprising a first fully connected layer, an activation layer, and a logistic regression layer; the optimization layer being used to determine the attention vector of the feature vector using a self-attention mechanism; The output layer includes a calculation layer, a second fully connected layer and a conditional random field decoding layer, wherein: The computing layer is used to assign weights to the feature vectors by multiplying the attention vector by the feature vector to obtain an optimized feature vector; The second fully connected layer and the conditional random field decoding layer are used to determine the labeling information for each character in the character sequence using a conditional random field model according to the optimized feature vector.

8. The method according to claim 6, wherein: The processing layer includes a first bidirectional long short-term memory network layer and a second bidirectional long short-term memory network layer; wherein: The first bidirectional long short-term memory network layer is used to take the first input vector as input and output a first feature vector; The second bidirectional long short-term memory network layer is used to take the second input vector as input and output a second feature vector. The feature vector of the character sequence is obtained by combining the first feature vector and the second feature vector.

9. A computer system comprising: one or more processors; a storage device for storing one or more programs, Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute: The text annotation method according to any one of claims 1 to 5; and / or A method for constructing a neural network model according to any one of claims 6 to 8.

10. A computer-readable storage medium having executable instructions stored thereon, which instructions, when executed by a processor, cause the processor to perform: The text annotation method according to any one of claims 1 to 5; and / or A method for constructing a neural network model according to any one of claims 6 to 8.

Images