Method, device and electronic device for determining word representation vectors

Abstract

Embodiments of the present application provide a method and device for determining word representation vectors, electronic equipment and computer readable storage medium, and belong to the field of natural language processing. The method comprises: obtaining a set of word units of a text; obtaining a context vector of the text based on the set of word units; and predicting a next word of the text based on the context vector. The method for determining word representation vectors can effectively obtain a corresponding set of word units even for pictographic characters or languages evolved from pictographic characters that are prone to out-of-vocabulary words, thereby improving the accuracy of determining word representation vectors.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and more specifically, to a method, apparatus, and electronic device for determining word representation vectors. Background Technology

[0002] In the field of natural language processing, determining word representation vectors to obtain the semantics of words has received widespread attention in recent years. Determining word representation vectors can be used to achieve various tasks in natural language processing, such as text summarization, machine translation, and image annotation, for example, predicting the next word given preceding text.

[0003] In determining word representation vectors, the word representation vectors are usually determined based on the character representation vectors of the characters in the given text. Therefore, it is necessary to optimize the existing methods for determining word representation vectors. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, electronic device, and computer-readable storage medium for determining word representation vectors.

[0005] In a first aspect, embodiments of this application provide a method for determining word representation vectors, the method comprising:

[0006] Obtain the set of glyph units for at least one word;

[0007] Based on the set of glyph units, obtain the word representation vector of the at least one word.

[0008] Secondly, embodiments of this application provide an apparatus for determining word representation vectors, the apparatus comprising:

[0009] The first acquisition module is used to acquire a set of glyph units for at least one word;

[0010] The second acquisition module is used to acquire the word representation vector of the at least one word based on the set of glyph units.

[0011] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory:

[0012] Memory, used to store machine-readable instructions;

[0013] A processor, configured to execute the method described in the first aspect when executing the machine-readable instructions described above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described in the first aspect above.

[0015] The beneficial effects of the technical solution provided in this application are:

[0016] The method described above for determining word representation vectors obtains word representation vectors through the set of glyph units in the text. Even for pictographic characters that are prone to having characters beyond the scope of the text, or languages ​​that have evolved from pictographic characters, it can effectively obtain the corresponding set of glyph units, thereby improving the accuracy of determining word representation vectors.

[0017] Furthermore, a set of glyph units is introduced. By sequentially extracting and fusing features at the glyph unit, character, and word levels, the final word representation vector includes richer semantic features, thereby improving the accuracy of the word representation vector.

[0018] Furthermore, for characters or words that are difficult to find, they are broken down into glyph units. The resulting glyph units can still contain valid semantic information, thereby further improving the accuracy of determining word representation vectors.

[0019] Furthermore, when fusing the first character feature vector and the second character feature vector, or when fusing the first word feature vector and the second word feature vector, we first obtain the gating weights through linear mapping, or obtain the feature weights of each character using cosine similarity. Then, we obtain the weighted sum of the first character feature vector and the second character feature vector, or the weighted sum of the first word feature vector and the second word feature vector, based on the gating weights or the character feature weights. This can effectively obtain the semantic information most relevant to the meaning of the character or word, thereby further improving the accuracy of determining the word representation vector. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0021] Figure 1 A schematic diagram of the structure of a neural network language model in the prior art is shown;

[0022] Figure 2 A schematic diagram of a deterministic word representation vector in a prior art neural network language model based on charCNN is shown.

[0023] Figure 3 This diagram illustrates an example of a prior art technique using a neural network language model based on charCNN to process characters beyond their syllabus.

[0024] Figure 4a The diagram illustrates how a prior art example of a charCNN-based neural network language model processes English words and Chinese words respectively.

[0025] Figure 4bThis diagram illustrates a prior art example of a neural network language model based on charCNN processing words.

[0026] Figure 5 This diagram illustrates a prior art example of a neural network language model based on charCNN processing words.

[0027] Figure 6 A flowchart illustrating a method for determining word representation vectors according to an embodiment of this application is shown;

[0028] Figure 7 A flowchart illustrating a method for determining word representation vectors according to an embodiment of this application is shown;

[0029] Figure 8 This illustration shows a schematic diagram of querying the feature vector of a character unit based on a character unit table in an example of an embodiment of this application;

[0030] Figure 9a The diagram illustrates a flowchart of obtaining word representation vectors for words in an example of an embodiment of this application.

[0031] Figure 9b A flowchart illustrating a method for determining word representation vectors according to an embodiment of this application is shown;

[0032] Figure 10 The diagram illustrates a flowchart of obtaining word representation vectors for words in an example of an embodiment of this application.

[0033] Figure 11 The diagram illustrates a flowchart of obtaining word representation vectors for words in an example of an embodiment of this application.

[0034] Figure 12 This document illustrates a flowchart of an example embodiment of the present application that uses the ABFE method to obtain the second character feature vector of a character;

[0035] Figure 13 This document illustrates a flowchart of an example embodiment of the present application that uses the ABFE method to obtain the second word feature vector of a word;

[0036] Figure 14 This document illustrates a flowchart of an example embodiment of the present application that uses the CosFE method to obtain the second character feature vector of a character;

[0037] Figure 15 This document illustrates a flowchart of an example embodiment of the present application that uses the GFF method to obtain the character representation vector of a character;

[0038] Figure 16The diagram illustrates a flowchart of obtaining word representation vectors using the GFF method in an example of an embodiment of this application.

[0039] Figure 17 A flowchart illustrating a method for determining word representation vectors in an example of an embodiment of this application is shown;

[0040] Figure 18 A flowchart illustrating a method for determining word representation vectors in an example of an embodiment of this application is shown;

[0041] Figure 19 This illustration shows a flowchart of text processing in one example of an embodiment of this application;

[0042] Figure 20 A flowchart illustrating a method for determining word representation vectors in an example of an embodiment of this application is shown;

[0043] Figure 21 A flowchart illustrating a method for determining word representation vectors in an example of an embodiment of this application is shown;

[0044] Figure 22a A schematic diagram of a scheme for determining word representation vectors for Latin words is shown in one example of this application;

[0045] Figure 22b The flowchart illustrating the method for determining word representation vectors in this application when applied to an input method is shown.

[0046] Figure 23 A schematic diagram of the interface when the method for determining word representation vectors of this application is applied to an input method is shown;

[0047] Figure 24 A comparison diagram showing the effect of the method for determining word representation vectors in this application with existing technologies is provided.

[0048] Figure 25 A comparison diagram showing the effect of the method for determining word representation vectors in this application with existing technologies is provided.

[0049] Figure 26 This diagram illustrates the structure of a device for determining word representation vectors according to an embodiment of this application.

[0050] Figure 27 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0051] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting the invention.

[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0053] To better understand the solutions provided in the embodiments of this application, the relevant technologies involved in this application will be briefly described below.

[0054] Application scenario: Input method

[0055] Pictographic writing and languages ​​derived from it, including Chinese and Japanese, are widely used worldwide, as are pictographic input methods.

[0056] Language models are a crucial module in input methods. They can predict the next word a user will type, correct typos, and list candidate sentences when the input sequence is ready. These functionalities can be improved by enhancing the performance of the language model. A superior language model makes the input method more user-friendly. A language model deployed on the terminal device can provide users with fast input responses, effectively protect user privacy, and save on network communication costs, thereby improving the user experience.

[0057] Statistical language models calculate the probability of the next word based on statistical information from a corpus.

[0058] like Figure 1As shown, a neural network language model mainly consists of three parts: a word representation module, an encoder, and an output layer. The input to the language model is a sequence of at least one word (also called an input word, input phrase, input sentence, or input statement). The language model needs to predict the word that may appear in the next position of the input word sequence.

[0059] The word representation module provides a word representation vector for each input word. The encoder processes these word representation vectors and generates a context vector. Based on the context vector, the output layer can predict what the next word will be. Therefore, the word representation module is a key module in neural language models.

[0060] If the word representation module fails to accurately represent the semantics, the context vector generated by the encoder will lose some important information. This leads to the neural language model predicting incorrect words regardless of the performance of the encoder and output layer. Therefore, the performance of a neural language model largely depends on the word representation module.

[0061] (1) In traditional feedforward neural network language models, the word representation module is a simple lookup operation. This operation searches for the corresponding word representation vector in a pre-stored vocabulary based on the input word. Its encoder is a feedforward neural network that encodes the sequence of word representation vectors corresponding to the input word sequence into a fixed-length context vector. The output layer takes the context vector generated by the encoder as input and calculates the probability of each word in the vocabulary appearing in the next position through linear operations and normalization. This then predicts the word in the next position.

[0062] (2) The recurrent neural network (RNN) language model has the same word representation module as the feedforward neural network model, which is a simple lookup table to obtain the word representation vector corresponding to the current word. Its encoder is a recurrent neural network. Compared with the feedforward neural network, the RNN has a more powerful encoding capability, and the context vector it encodes is of higher quality than the context vector obtained by the feedforward neural network. The output layer of the RNN is the same as that of the feedforward neural network language model.

[0063] (3) Neural network language models based on charCNN (Character-level Convolutional Networks), such as Figure 2As shown, the word representation module of this model is a charCNN network. In this language model, the input words are first split into characters. Through a look-up table operation (an operation to look up the character table), the corresponding character representation vectors for each character are found, and then through a concatenation operation, the character representation vectors corresponding to all the characters in a word are concatenated into a feature matrix. The charCNN network extracts some intermediate features through convolutional operations on this feature matrix. Finally, pooling operations are performed on these intermediate features to extract a feature with a fixed length, which is the word representation vector corresponding to the input word. The encoder of this language model is a recurrent neural network, such as an LSTM (Long Short-term Memory) network. The encoder uses the word representation vector as input and outputs a context vector. The output layer is the same as that of a feed-forward neural network language model. The output layer is a fully connected layer that obtains the conditional probability of the next word appearance based on the context vector.

[0064] 1) When the input word is an out-of-vocabulary word not stored in the word list, the word representation modules of the feed-forward neural network language model and the recurrent neural network language model cannot generate effective word representation vectors, thus affecting the performance of the entire language model. As Figure 3 shown, the word representation module of the neural network language model based on charCNN can calculate effective word representation vectors based on the character representation vectors, thus solving the problem of out-of-vocabulary words to a certain extent. However, when the input word consists of out-of-character-table out-of-character characters, the word representation module of the neural network language model based on charCNN also cannot generate accurate and effective word representation vectors, that is Figure 3 as shown in, an invalid character-word representation is obtained. However, for hieroglyphic languages (such as Chinese), the number of single characters is very large (more than 91,000 single characters in Chinese). In this case, it is inevitable to have out-of-character characters. Therefore, when dealing with hieroglyphic languages, the charCNN-based language model will have the problem of being unable to handle out-of-vocabulary words composed of out-of-character characters.

[0065] Figure 3 In, for example, for the word "jiān jiā", when this word is out-of-vocabulary, the look-up table operation will return an invalid feature vector because the feature vectors corresponding to the characters of this word cannot be found in the character table, and the concatenated matrix is an invalid matrix. After convolutional operations, invalid character-level features are obtained, and finally Figure 3 as shown in, an invalid character-word representation is obtained, that is, an invalid word representation vector. The charCNN outputs an invalid word representation form. At this time, the language model often cannot correctly predict the next word because it cannot obtain any useful information from the word representation. In addition to "jiān jiā", there are many similar out-of-vocabulary words, such as "yīn yì", etc.

[0066] 2) The word representation module of the language model based on charCNN generates corresponding word representation vectors through convolution operations. For hieroglyphic languages such as Chinese, the length of words is very short (usually only one or two characters), and convolution operations are not very suitable for processing such short sequences. In addition, for hieroglyphic languages, charCNN extracts semantics from characters. However, for hieroglyphic languages such as Chinese, its characters already have rich semantic information. In this case, it is not necessary to use charCNN to extract semantic information from combinations of characters.

[0067] As Figure 4a shown, most hieroglyphs are very short, but charCNN aims to process long sequences like English words. For example, the English word "preprocess" consists of 10 characters, while the Chinese word "预处理" (preprocess) consists of only 3 characters. Therefore, many feature spaces in the concatenated feature matrix are not used, resulting in a waste of resources.

[0068] The convolution operation in charCNN aims to extract high-level features from combinations of basic elements. However, different from English characters, hieroglyphs are already high-level features. For example, the embedding of the character "预" already contains the meaning of "预处理". For English, however, it is necessary to extract the meaning of "precede" from "pre" through convolution operations.

[0069] 3) For hieroglyphic languages or languages evolved from hieroglyphs, such as Chinese, each single character is composed of some independent components - glyph units (which can be called characters), and some glyph units themselves also have rich semantic information. However, none of the above language models take this into account.

[0070] 4) For hieroglyphic languages or languages evolved from hieroglyphs such as Chinese, the contribution of each single character in a word to the word meaning is different (for example, in the word "斗牛犬", the character "犬" has the greatest semantic contribution). However, none of the above language models perform special processing on this language phenomenon.

[0071] As Figure 4b shown, the contributions of characters in a word are different semantically, but charCNN does not treat them differently. For example, "犬" (dog) has a greater semantic contribution to the word "斗牛犬" (bulldog) than "斗" (fight), but charCNN processes them using the same operation.

[0072] Therefore, the word representation module of the language model based on charCNN cannot accurately encode the semantic information of words.

[0073] When the input word consists of characters not included in the word list, or when at least one character in the word is not included in the word list, charCNN will output an invalid word representation vector, which may not contain any useful information.

[0074] For some pictographic scripts, it's unavoidable that certain characters are not included in the character list; for example, the number of Chinese characters exceeds 91,000. Due to limited storage space in terminal devices (such as mobile phones, tablets, etc.), the size of the character list cannot be too large. There are always new characters on social networks, many of which are newly created and therefore cannot be found in any existing character list; some characters are rare, and there isn't enough data to train them, so they are often excluded from the character list.

[0075] like Figure 5 As shown, when the input word consists of characters not included in the character list (i.e., the word table), the lookup table operation will return an invalid feature vector (also called an invalid feature embedding) because the feature vector corresponding to the word cannot be found in the word table. The concatenated matrix is ​​an invalid matrix, and after convolution, invalid character-level features (also called invalid character feature vectors) are obtained, ultimately leading to... Figure 5 The invalid word representation vector shown is an invalid word representation. Because charCNN outputs invalid word representation vectors, the language model often cannot correctly predict the next word, as it cannot obtain any useful information from the vocabulary. If some characters in the input word are not included in the vocabulary, then the word representation vectors of the characters not in the vocabulary are invalid feature vectors, resulting in inaccurate word representation vectors in the final product.

[0076] In conclusion, we can see that language models based on charCNN are not suitable for pictographic languages.

[0077] To address the aforementioned problems, this invention proposes the following solution: a neural language model based on multi-level word representation, which is more suitable for processing pictographic characters like Chinese.

[0078] 1. The language model in this invention takes into account the glyph units that make up a single character and incorporates these glyph units into the word representation module. This improves the quality of the word representation vectors generated by the module and solves the problem of invalid word representation vectors generated by the word representation module due to out-of-syllabus words composed of out-of-syllabus characters.

[0079] 2. The language model proposed here has a word representation module that adopts a hierarchical network structure to achieve multi-level word representations. It uses a feature fusion method based on gated units to fuse the features at the glyph unit, character, and word levels layer by layer. This network structure is consistent with the composition method of words in hieroglyphic languages (where glyph units form single characters, and then single characters form words). Therefore, compared with the word representation module based on charCNN, the multi-level word representation module proposed in this language model is more suitable for processing hieroglyphic languages such as Chinese.

[0080] 3. For hieroglyphic languages or languages evolved from hieroglyphics, some glyph units themselves also have rich semantic information. However, none of the above language models have considered this problem, and the solution to this problem is the same as in 1.

[0081] 4. In the word representation module of this language model, a feature extraction method based on the attention mechanism is used. This method can extract the information of those characters in the word that are more relevant to the word meaning from the characters that make up the word, weakening or even ignoring the irrelevant characters (for example, in the word "bulldog", this extraction method will focus on extracting the semantic information contained in the character "dog", while appropriately weakening the information of the characters "fight" and "bull").

[0082] The objective of the present invention: A multi-level word representation method introducing glyph unit information is proposed. By introducing glyph unit information and extracting the features of glyph units or successively extracting and fusing the features at the glyph unit, character, and word levels, the quality of the word representation vectors generated by the word representation module is improved, thereby enhancing the performance of the entire language model.

[0083] The technical solution of the present application and how the technical solution of the present application solves the above technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0084] Figure 6 The flowchart of a method for determining word representation vectors provided by an embodiment of the present application is shown, as Figure 6 shown in, the method may include the following steps:

[0085] Step S601, obtain a set of glyph units of at least one word.

[0086] In specific implementation, at least one word can be obtained. At least one word includes at least one character, and one character includes at least one glyph unit.

[0087] The glyph unit in the present application refers to the basic unit with semantic meaning in the glyph structure of a character in the text.

[0088] In the present application, each character is correspondingly provided with a fixed splitting method. Based on the fixed splitting method of each character, all the glyph structures of at least one character can be split to obtain the corresponding at least one glyph unit.

[0089] Taking the character "进" as an example, the character "进" has only one splitting method, obtaining "辶" and "井", and the entire character "进" is completely split without any remaining part.

[0090] At least one character of at least one word can be split to obtain a set of glyph units of at least one word. A kind of text can be correspondingly split to obtain a set of glyph units, and the set of glyph units includes at least one glyph unit.

[0091] Step S602: Based on the set of glyph units, obtain the word representation vector of the at least one word.

[0092] Among them, the word representation vector is a vector used to represent the semantics of the corresponding word. In this embodiment, based on the set of glyph units, a glyph unit feature vector corresponding to the set of glyph units can be obtained, and based on the glyph unit feature vector, the corresponding word representation vector can be obtained.

[0093] In other embodiments, the word representation vector can be obtained by combining the glyph unit feature vector and the first character feature vector of the corresponding character; the word representation vector can also be obtained by combining the word representation vector and the first word feature vector of the corresponding word. The specific method for obtaining the word representation vector will be elaborated in detail below.

[0094] For the above method of determining the word representation vector, the word representation vector is obtained through the set of glyph units of the text. Even for hieroglyphics that are prone to contain out-of-range characters or languages evolved from hieroglyphics, the corresponding set of glyph units can be effectively obtained, thereby improving the accuracy of determining the word representation vector.

[0095] A possible implementation manner of the embodiment of the present application. The obtaining of the set of glyph units of at least one word in step S601 may include:

[0096] (1) Split the at least one word to obtain at least one character;

[0097] (2) Split the at least one character obtained by splitting to obtain at least one glyph unit, and obtain the set of glyph units based on the at least one glyph unit obtained.

[0098] Specifically, a preset network model can be used to split at least one word into at least one character, and then the character obtained by splitting is split according to the character table to obtain each glyph unit constituting the character.

[0099] For example, the word "flowers and plants" can be split into two characters "flower" and "grass". For the character "flower", it can be split into two glyph units "艹" and "hua". For the character "grass", it can be split into two glyph units "艹" and "zao".

[0100] Another example is that the word "jianjia" can be split into two characters "jian" and "jia". For the character "jian", it can be split into two glyph units "艹" and "jian". For the character "jia", it can be split into two glyph units "艹" and "jia". Even though the two characters "jianjia" are out-of-vocabulary words, the split "艹" still contains valid semantic information: "jianjia" refers to a kind of herbaceous plant, and "艹" includes the meaning of herbaceous plants.

[0101] In the specific implementation process, various different glyph units are set in the character table.

[0102] In the above embodiments, by setting a glyph unit table, the glyph unit table includes all glyph units that make up Chinese characters. For any word, splitting it into glyph units, regardless of whether the characters in the word are out-of-vocabulary words or difficult to query, the glyph units that make up the word can always be found in the glyph unit table, and the obtained glyph units can still include valid semantic information, thereby improving the accuracy of determining the word representation vector.

[0103] A possible implementation manner of the embodiment of the present application, as Figure 7 shown, obtaining the word representation vector of at least one word based on the glyph unit set in step S602 may include:

[0104] Step S610, obtaining the glyph unit feature vector of the glyph unit set;

[0105] Step S620, obtaining the word representation vector of at least one word based on the glyph unit feature vector.

[0106] Specifically, for each glyph unit in the glyph unit table, a corresponding glyph unit feature vector can be preset, and the glyph unit feature vectors corresponding to at least one glyph unit in the glyph unit set can be directly queried.

[0107] Specifically, a glyph unit table can be set, as Figure 8 shown, that is, Figure 8 the glyph unit table in, there are multiple glyph units set in the glyph unit table, and the glyph unit feature vector corresponding to each glyph unit, taking Figure 8 the "grass" shown in as an example, "grass" includes two glyph units "艹" and "zao", and the glyph unit feature vector corresponding to "艹" can be queried, as well as the glyph unit feature vector corresponding to "zao".

[0108] Specifically, one can directly obtain the character representation vector of at least one character corresponding to the set of character units based on the character unit feature vector, and then obtain the word representation vector based on the character representation vector; alternatively, one can query the first character feature vector of at least one character corresponding to the set of character units, obtain the character representation vector based on the first character feature vector and the character unit feature vector, and then obtain the word representation vector based on the character representation vector; alternatively, after obtaining the character representation vector, one can obtain the first word feature vector of the corresponding word, and determine the second word feature vector of the corresponding word based on the character representation vector, and finally obtain the final word representation vector based on the first word feature vector and the second word feature vector. The specific process of obtaining the word representation vector will be explained in detail below.

[0109] The following will further illustrate the specific process of obtaining word representation vectors with reference to the accompanying drawings and specific embodiments.

[0110] In one implementation, step S620, which involves obtaining the word representation vector of at least one word based on the glyph unit feature vector, may include:

[0111] (1) Based on the feature vector of the character unit, obtain the character representation vector corresponding to at least one character in at least one word.

[0112] Specifically, for any character in at least one word, at least one glyph unit that makes up that character can be obtained, and then the glyph unit feature vector that makes up that character can be determined. By weighted summing of the glyph unit feature vectors of at least one glyph unit that makes up that character, the character representation vector of that character can be obtained.

[0113] (2) Based on the character representation vector corresponding to at least one character in at least one word, obtain the word representation vector of at least one word.

[0114] Specifically, for any word in at least one word, the word representation vector of the word can be obtained by weighted summing of the character representation vectors of at least one character in that word.

[0115] like Figure 9a As shown, for at least one word, the word is split into at least one character, and the at least one character is split into a set of glyph units, thereby obtaining the set of glyph units of the at least one word, and obtaining the corresponding word representation vector based on the set of glyph units.

[0116] In another implementation, obtaining the character representation vector corresponding to at least one character in at least one word based on the glyph unit feature vector may include:

[0117] a. Obtain the first character feature vector of at least one character in the at least one word;

[0118] Specifically, for any character in a pre-set character table, a corresponding first character feature vector can be set, and the first character feature vectors respectively corresponding to at least one character in at least one word can be directly queried.

[0119] In a specific implementation process, obtaining the first character feature vectors of at least one character in the at least one word may include:

[0120] a1. Query the first character feature vectors corresponding to at least one character in the at least one word;

[0121] a2. If the corresponding first character feature vectors are not queried, set the preset character feature vector as the first character feature vector.

[0122] Among them, the preset character feature vector may be a vector set for all out-of-vocabulary characters for which the corresponding first character feature vectors cannot be queried. The preset character feature vector may be a vector used to represent all out-of-vocabulary characters. For example, it is represented by "unknown character feature vector". If the corresponding first character feature vectors are not queried, the preset character feature vector is used as the first character feature vector of this character.

[0123] b. Obtain the second character feature vectors corresponding to at least one character in the at least one word according to the glyph unit feature vectors.

[0124] Specifically, the glyph unit weight coefficients of at least one glyph unit forming a character can be determined, and the corresponding glyph unit feature vectors are fused according to the glyph unit weight coefficients to obtain the corresponding second character feature vector.

[0125] For example, for a character "进" (jìn), which includes two glyph units "辶" (zǒu) and "井" (jǐng), the glyph unit feature vectors corresponding to "辶" and "井" can be obtained respectively, and then the two glyph unit feature vectors are weighted and summed to obtain the second character feature vector of "进".

[0126] Obtaining the second character feature vectors corresponding to at least one character in the at least one word according to the glyph unit feature vectors may include:

[0127] b1. Determine the glyph unit weight coefficients corresponding to at least one character in the at least one word.

[0128] Specifically, determining the glyph unit weight coefficients corresponding to at least one character in the at least one word may include:

[0129] ① Determine the inner product of the glyph unit feature vectors of the glyph units of at least one character and the corresponding first character feature vector of the character to obtain the initial glyph unit weight of the glyph unit;

[0130] ②Based on the initial glyph unit weights, determine the glyph unit weight coefficients corresponding to the said glyph units.

[0131] In a specific implementation process, the inner product of the glyph unit feature vector of a glyph unit and the first character feature vector of a character can be obtained, that is, the glyph unit feature vector and the first character feature vector are multiplied by the inner product to obtain the initial glyph unit weight of a glyph unit.

[0132] Taking "进" as an example, multiply the glyph unit feature vector of "辶" and the first character feature vector of "进" by the inner product to obtain the initial glyph unit weight of "辶", for example, obtaining 8; then multiply the glyph unit feature vector of "井" and the first character feature vector of "进" by the inner product to obtain the initial glyph unit weight of "井", for example, obtaining 2. According to the ratio of the initial glyph unit weights of "辶" and "井", calculate the glyph unit weight coefficients of "辶" and "井" respectively.

[0133] It can be understood that the sum of the glyph unit weight coefficients of multiple glyph units in a character is 1, that is, the respective initial glyph unit weights are normalized to obtain the character weight coefficient of "辶" as 0.8 and the character weight coefficient of "井" as 0.2.

[0134] In the above implementation manner, the initial glyph unit weight of the glyph unit is determined by the way of multiplying the inner product. In other implementation manners, other ways can also be used to determine the initial glyph unit weight of the glyph unit. For example, the way of product can be used, which is not limited herein.

[0135] b2. Based on the determined glyph unit weight coefficients and the corresponding glyph unit feature vectors, obtain the corresponding second character feature vector.

[0136] Specifically, according to the glyph unit weight coefficients of each glyph unit, perform weighted summation on the glyph unit feature vectors of each glyph unit to obtain the second character feature vector of this character.

[0137] Taking the character "进" as an example, multiply the glyph unit weight coefficient of "辶" by the glyph unit feature vector of "辶", multiply the glyph unit weight coefficient of "井" by the glyph unit feature vector of "井", and add the two glyph unit feature vectors after multiplying by the glyph unit weight coefficients to obtain the second character feature vector of "进".

[0138] c. According to the first character feature vector and the second character feature vector, obtain the character representation vector corresponding to the at least one character.

[0139] Specifically, the first character feature vector and the second character feature vector can be fused to obtain the character representation vector corresponding to the character. The specific process of determining the character representation vector will be elaborated in detail below.

[0140] A possible implementation manner of the embodiment of the present application for obtaining the character representation vector corresponding to at least one character according to the first character feature vector and the second character feature vector may include:

[0141] c1. Perform a linear mapping calculation on the first character feature vector to obtain the first character weight coefficient of the first character feature vector;

[0142] c2. Obtain the second character weight coefficient of the second character feature vector based on the first character weight coefficient;

[0143] c3. Obtain the character representation vector corresponding to the character according to the first character weight coefficient, the first character feature vector, the second character weight coefficient, and the second character feature vector.

[0144] Taking the character "进" as an example, the first character feature vector of "进" can be queried first, and then the second character feature vector of "进" can be obtained according to the glyph unit feature vectors of "辶" and "井"; perform a linear mapping calculation based on the queried first character feature vector to obtain the first character weight coefficient, then calculate the second character weight coefficient according to the first character weight coefficient, and finally calculate the character representation vector of "进".

[0145] The specific process of the linear mapping will be elaborated in detail below.

[0146] As Figure 9b shown, taking a word including one character as an example, the glyph unit set of the character can be obtained, and the second character feature vector of the character can be obtained according to the glyph unit set; query the first character feature vector of the character, and determine the word representation vector of the word according to the first character feature vector and the second character feature vector. That is to say, the final word representation vector is determined according to the glyph unit set and the first character feature vector.

[0147] In another implementation manner, for obtaining the word representation vector of at least one word based on the character representation vector corresponding to at least one character in at least one word, it may include:

[0148] e. Obtain the first word feature vector of at least one word.

[0149] Specifically, for any word in the pre-set word list, a corresponding first word feature vector can be set, and directly query the first word feature vectors corresponding to at least one word respectively.

[0150] In the specific implementation process, obtaining the first word feature vector of at least one word may include:

[0151] e1. Query the first word feature vector corresponding to the at least one word;

[0152] e2. If the corresponding first word feature vector is not queried, set the preset word feature vector as the first word feature vector.

[0153] Among them, the preset word feature vector can be a vector set for all out-of-vocabulary words for which the corresponding first word feature vector cannot be queried. The preset word feature vector can be a vector used to represent all out-of-vocabulary words. For example, it is represented by "unknown word feature vector". If the corresponding first word feature vector is not queried, use the preset word feature vector as the first word feature vector of this character.

[0154] f. Based on the character representation vectors corresponding to the at least one character, obtain the second word feature vector of the at least one word.

[0155] Specifically, the word weight coefficients of at least one character constituting a word can be determined, and the corresponding character representation vectors are fused according to the word weight coefficients to obtain the corresponding second word feature vector.

[0156] Specifically, obtaining the second word feature vector of the at least one word based on the character representation vectors corresponding to the at least one character may include:

[0157] f1. Determine the inner product of the character representation vector of at least one character in the at least one word and the first word feature vector of the corresponding word to obtain the initial word weight of at least one character;

[0158] f2. Based on the initial word weight, obtain the word weight coefficient of at least one character;

[0159] f3. Based on the word weight coefficients of at least one character and the corresponding character representation vectors, obtain the second word feature vector of the corresponding word.

[0160] Specifically, obtain the inner product of the character representation vector of a character and the first word feature vector of a word, that is, multiply the character representation vector and the first word feature vector by the inner product to obtain the initial word weight of a character representation vector.

[0161] Taking "jianjia" as an example, multiply the character representation vector of "jian" and the first word feature vector of "jianjia" by the inner product to obtain the initial word weight of "jian", for example, 10; then multiply the character representation vector of "jia" and the first word feature vector of "jianjia" by the inner product to obtain the initial word weight of "jia", for example, 30. According to the ratio of the initial word weights of "jian" and "jia", calculate the word weight coefficients of "jian" and "jia" respectively.

[0162] In the above embodiments, the initial word weight of a character is determined by inner product multiplication. In other embodiments, other methods may also be used to determine the initial word weight of a character. For example, the product method may be used, which is not limited herein.

[0163] It can be understood that the sum of the word weight coefficients of multiple characters in a word is 1, that is, the respective initial word weights are normalized to obtain a word weight coefficient of 0.25 for "蒹" and a word weight coefficient of 0.75 for "葭".

[0164] g. Obtain the word representation vector of the at least one word according to the first word feature vector and the second word feature vector.

[0165] Specifically, the first word feature vector and the second word feature vector can be fused to obtain the word representation vector of the corresponding character.

[0166] Specifically, obtaining the word representation vector of the at least one word according to the first word feature vector and the second word feature vector may include:

[0167] g1. Perform a linear mapping calculation on the first word feature vector to obtain the first word weight coefficient of the first word feature vector;

[0168] g2. Obtain the second word weight coefficient of the second word feature vector based on the first word weight coefficient;

[0169] g3. Obtain the word representation vector of the corresponding word according to the first word weight coefficient, the first word feature vector, the second word weight coefficient, and the second word feature vector.

[0170] Taking the word "蒹葭" as an example, the first word feature vector of "蒹葭" can be queried first, and then the second word feature vector of "蒹葭" can be obtained according to the character feature vectors of "蒹" and "葭"; perform a linear mapping calculation based on the queried first word feature vector to obtain the first word weight coefficient, then calculate the second word weight coefficient according to the first word weight coefficient, and finally calculate the word representation vector of "蒹葭".

[0171] Such as Figure 10 As shown, taking a word including one character as an example, the glyph unit set of the character can be obtained, the word representation vector of the character can be obtained according to the glyph unit set, and the second word feature vector of the word can be determined based on the word representation vector; query the first word feature vector of the word, and determine the word representation vector of the word based on the first word feature vector and the second word feature vector. That is, the final word representation vector is determined based on the glyph unit and the first word feature vector.

[0172] Such as Figure 11As shown, taking a word that includes a single character as an example, we can obtain the set of character glyph units, and obtain the second character feature vector of the character based on the set of character glyph units; query the first character feature vector of the character, and determine the second word feature vector of the word based on the first character feature vector and the second character feature vector; query the first word feature vector of the word, and determine the word representation vector of the word based on the first word feature vector and the second word feature vector. In other words, the final word representation vector is determined based on the character glyph units, the first character feature vector, and the first word feature vector.

[0173] In the above embodiments, a set of glyph units is introduced. By sequentially extracting and fusing features at the three levels of glyph units, characters, and words, the final word representation vector includes richer semantic features, thereby improving the accuracy of the word representation vector.

[0174] The above embodiments illustrate the specific acquisition process of the second character feature vector, character representation vector, second word feature vector, and word representation vector. The following will further elaborate on the process of obtaining the character unit weight coefficient or character weight coefficient by multiplying the inner product and normalizing it, in conjunction with specific embodiments and accompanying drawings.

[0175] One possible implementation of this application embodiment is to use the Attention-based Feature Extractor (ABFE) method to obtain the second character feature vector of a character based on the character unit vector of each character unit, or to obtain the second word feature vector of a word based on the character feature vector of each character unit.

[0176] The ABFE method, through all low-level features... i A weighted average summation operation is performed to obtain the extracted high-level feature vector h.

[0177]

[0178] In the formula: l i h represents the i-th low-level feature; h represents the high-level feature; w i This represents the weight coefficient of the i-th low-level feature; i is a natural number.

[0179] It is understandable that if l i Let h be the character representation vector corresponding to the i-th character in a word, and w be the second word feature vector of a word. i That is, the word weight coefficient of the i-th word; if l i Let h be the feature vector of the i-th glyph unit in a character, and w be the feature vector of the second glyph unit in a character. i This represents the weight coefficient of the i-th character unit.

[0180] The weights in the weighted average summation operation in equation (1) above, i.e., the weight coefficients of the character units, are calculated through the attention mechanism:

[0181] w i =softmax(l i T h emb (2)

[0182] In the formula, h emb For high-level feature vectors, if l i h is the character representation vector corresponding to the i-th character in a word. emb That is, representing the first word feature vector of a word; if l i h is the feature vector of the glyph unit corresponding to the i-th glyph unit in a character. emb This represents the first character feature vector of a character.

[0183] Low-level features l i and high-level feature vector h emb The inner product operation is performed between them to obtain the unnormalized weights, i.e., the initial glyph unit weights, or initial word weights. These weights are then normalized using a Softmax operation, which makes the glyph unit weight coefficients or word weight coefficients w... i The sum of them is one.

[0184] In this embodiment, the low-level feature l i and high-level feature vector h emb The inner product operation is performed to obtain the unnormalized weights. In other implementations, the unnormalized weights can also be determined in other ways, such as by product, which is not limited here.

[0185] When l i and h emb When the dimensions are different, the specific calculation process is as follows:

[0186] w i =softmax(l i T Wh emb (3)

[0187] First use matrix W on h emb Perform a linear mapping such that its dimension is the same as l. i To maintain consistency, perform inner product and Softmax operations to obtain the weights for the weighted summation.

[0188] like Figure 12As shown, taking the character "树" as an example, the process of obtaining the second character feature vector of "树" (i.e., the character-level feature vector shown in the figure) using the ABFE method is as follows: Obtain the first character feature vector of "树", which is the "character feature vector of 树" shown in the figure; separately obtain the glyph units "木", "又", and "寸" of "树", and then separately query the glyph unit feature vectors of "木", "又", and "寸", which are the feature vectors of the glyph units shown in the figure. Multiply the glyph unit feature vector of "木" by the first character feature vector of "树" through inner product to obtain the initial glyph unit weight of "木"; multiply the glyph unit feature vector of "又" by the first character feature vector of "树" through inner product to obtain the initial glyph unit weight of "又"; multiply the glyph unit feature vector of "寸" by the first character feature vector of "树" through inner product to obtain the initial glyph unit weight of "寸"; normalize the initial glyph unit weights of "木", "又", and "寸" to obtain the glyph unit weight coefficients of "木", "又", and "寸", which are 0.9, 0.05, and 0.05 respectively. Then, based on each glyph unit weight coefficient, obtain the weighted sum of each glyph unit feature vector to obtain the second character feature vector of "树", which is the "character-level feature vector of 树" shown in the figure. Among them, the above normalization operation can be the Softmax operation.

[0189] Similarly, as Figure 13 shown, taking the word "斗牛犬" as an example, the process of obtaining the second word feature vector of "斗牛犬" (i.e., the word-level feature vector shown in the figure) using the ABFE method is as follows: Obtain the first word feature vector of "斗牛犬", which is the "word feature vector of 斗牛犬" shown in the figure; separately obtain the character representation vectors of "斗", "牛", and "犬" in "斗牛犬", and then multiply the character representation vector of "斗" by the first character feature vector of "斗牛犬" through inner product to obtain the initial character weight of "斗"; multiply the character representation vector of "牛" by the first word feature vector of "斗牛犬" through inner product to obtain the initial character weight of "牛"; multiply the character representation vector of "犬" by the first word feature vector of "斗牛犬" through inner product to obtain the initial character weight of "犬"; normalize the initial character weights of "斗", "牛", and "犬" to obtain the character weight coefficients of "斗", "牛", and "犬", which are 0.1, 0.2, and 0.7 respectively. Then, based on each character weight coefficient, obtain the weighted sum of each character feature vector to obtain the second word feature vector of "斗牛犬", which is the "word-level feature vector of 斗牛犬" shown in the figure.

[0190] The higher the semantic contribution of a character or glyph unit, the greater the corresponding weight. In this case, the proportion of "犬" is the largest. Using the Softmax operation can ensure that the sum of the weights is 1. Characters or glyph units with more semantic contribution degrees are set with larger weights, and more information related to the semantics of the word can be extracted.

[0191] The above embodiments describe the process of obtaining the glyph unit weight coefficient or word weight coefficient through inner product multiplication and normalization, that is, the specific process of the ABFE method.

[0192] In a possible implementation manner of the embodiments of the present application, a cosine-based feature extractor (CosFE) method may also be adopted to obtain a second word feature vector of a word based on the glyph unit vectors of each glyph unit, or obtain a second word feature vector of a word based on each word feature vector.

[0193] The process of obtaining a second word feature vector of a word based on the glyph unit vectors of each glyph unit by using the CosFE method is as follows:

[0194] Obtain the cosine similarity between the glyph unit feature vector of each glyph unit and the first word feature vector of a word, and use the obtained values of each cosine similarity as the corresponding glyph unit weight coefficient. It should be noted that the glyph unit weight coefficient here does not need to be normalized again, and the sum of each glyph unit weight coefficient may not be 1. Obtain the weighted sum based on each glyph unit weight coefficient and the corresponding glyph unit feature vector to obtain the second word feature vector.

[0195] As Figure 14 shown, taking the example of obtaining the second word feature vector of "brick", that is, the "word-level feature vector of brick" shown in the figure, respectively obtain the glyph unit feature vectors of "stone" and "zhuan", that is, the feature vectors of the glyph units shown in the figure, and obtain the cosine similarity between the glyph unit feature vector of "stone" and the first word feature vector of "brick"; obtain the cosine similarity between the glyph unit feature vector of "zhuan" and the first word feature vector of "brick", and use the obtained similarity as the corresponding glyph unit weight coefficient, and obtain the weighted sum between each glyph unit feature vector to obtain the second word feature vector of "brick", that is, the "word-level feature vector of brick" shown in the figure.

[0196] It can be understood that in other embodiments, other methods may also be adopted to obtain a second word feature vector of a word based on the glyph unit vectors of each glyph unit, or obtain a second word feature vector of a word based on each word feature vector.

[0197] The following will further elaborate in detail the process of linear mapping and the fusion of the first word feature vector and the second word feature vector, or the fusion of the first word feature vector and the second word feature vector based on the result of the linear mapping, in combination with specific embodiments and drawings.

[0198] One possible implementation of this application embodiment is that the method of fusing the first character feature vector and the second character feature vector, as well as the method of fusing the first word feature vector and the second word feature vector, can both be described as a gated feature fusion (GFF) method.

[0199] GFF is used to fuse the extracted feature vector f ex (i.e., the feature vector of the second character or the feature vector of the second word) and the feature vector f obtained by looking up the table. emb (i.e., the feature vector of the first character or the feature vector of the first word), the extracted feature vector f ex and the eigenvector f obtained by looking up the table emb The proportion involved in the fusion process is determined by the gate value w. g control.

[0200] f rep =(1-w g )f emb +w g f ex (4)

[0201] In the formula: f ex f is the extracted low-level feature vector; emb f is the low-level feature vector obtained from the lookup table; rep The high-level feature vector obtained by fusion; w g This is the gate value.

[0202] In other words, during the process of obtaining the word representation vector, f ex f is the feature vector of the second character. emb f is the first character feature vector. rep f is a character representation vector; in the process of obtaining the word representation vector, f ex f is the feature vector of the second word. emb f is the feature vector of the first word. rep The word is represented by a vector.

[0203] Gating value w g The eigenvector f is obtained by GFF from the lookup table. emb The calculated gate value is a value between 0 and 1. The specific calculation process for the gate value is as follows:

[0204] w g =σ(ω) T f emb (5)

[0205] ω is a trainable weight vector, and σ(·) is a sigmoid function that makes the output w g It is a value between 0 and 1.

[0206] As Figure 15 shown, taking the example of obtaining the word representation vector of the character "rán", the first character feature vector of "rán" is obtained, which is the character feature vector of "rán" shown in the figure; the second character feature vector of "rán" is obtained, which is the extracted character-level feature vector corresponding to "rán" shown in the figure; based on the first character feature vector of the character "rán", a gating weight of 0.05 is obtained, 1 minus the gating weight is calculated to get the first character weight coefficient 0.95 corresponding to the first character feature vector, the first character feature vector is multiplied by the first character weight coefficient 0.95, and the second character feature vector is multiplied by the gating weight 0.05, and the two are added together to obtain the final word representation vector, which is the "fused character-level feature vector" shown in the figure.

[0207] Similarly, as Figure 16 shown, taking the example of obtaining the word representation vector of the word "chocolate", the first word feature vector of "chocolate" is obtained, which is the word feature vector of "chocolate" shown in the figure; the second word feature vector of "chocolate" is obtained, which is the extracted word-level feature vector corresponding to "chocolate" shown in the figure; based on the first word feature vector of the word "chocolate", a gating weight of 0.02 is obtained, 1 minus the gating weight is calculated to get the first word weight coefficient 0.98 corresponding to the first word feature vector, the first word feature vector is multiplied by the first word weight coefficient 0.98, and the second word feature vector is multiplied by the gating weight 0.02, and the two are added together to obtain the final word representation vector, which is the "fused word-level feature vector" shown in the figure.

[0208] By using the gating weight, GFF can adjust the ratio of the first word feature vector and the second word feature vector. For example, the characters "qiǎo", "kè", and "lì" do not contain any semantic meaning related to the word "chocolate". Therefore, the first word feature vector extracted from these characters is useless for the word representation of "chocolate", and its ratio is very small, as shown in the left figure as 0.05, while the ratio of the word embedding is larger, which is 0.95. In this way, the ratio with low semantic contribution will be very small, while the ratio with high semantic contribution will be very large, and the effective semantic features can be better retained.

[0209] In other embodiments, the GFF method can also be replaced by addition, that is, directly adding and fusing the first word feature vector and the second word feature vector.

[0210] In the above embodiments, when fusing the first character feature vector and the second character feature vector, or when fusing the first word feature vector and the second word feature vector, a gating weight is first obtained through linear mapping, and then a weighted sum of the first character feature vector and the second character feature vector, or a weighted sum of the first word feature vector and the second word feature vector, is obtained based on the gating weight. This can effectively obtain the semantic information most relevant to the meaning of the character or word, thereby further improving the accuracy of determining the word vector.

[0211] The method described above for determining word representation vectors obtains word representation vectors by using the set of glyph units in the text. Even for pictographic characters or languages ​​evolved from pictographs that are prone to having characters beyond the scope of the text, the method can effectively obtain the corresponding set of glyph units, thereby improving the accuracy of determining word representation vectors.

[0212] Furthermore, a set of glyph units is introduced. By sequentially extracting and fusing features at the glyph unit, character, and word levels, the final word representation vector includes richer semantic features, thereby improving the accuracy of the word representation vector.

[0213] Furthermore, for characters or words that are difficult to find, they are broken down into glyph units. The resulting glyph units can still contain valid semantic information, thereby further improving the accuracy of determining word representation vectors.

[0214] Furthermore, when fusing the first character feature vector and the second character feature vector, or when fusing the first word feature vector and the second word feature vector, we first obtain the gating weights through linear mapping, or obtain the feature weights of each character using cosine similarity. Then, we obtain the weighted sum of the first character feature vector and the second character feature vector, or the weighted sum of the first word feature vector and the second word feature vector, based on the gating weights or the character feature weights. This can effectively obtain the semantic information most relevant to the meaning of the character or word, thereby further improving the accuracy of determining the word representation vector.

[0215] To better understand the method for determining word representation vectors described above, an example of a determined word representation vector according to the present invention is described in detail below:

[0216] In one example, such as Figure 17 As shown, the method for determining word representation vectors provided in this application can be applied to... Figure 17 The device shown can be composed of three main parts: a Hierarchical Word Representation (HWR) module, an encoder, and an output layer.

[0217] 1. The first part is our newly proposed multi-level word representation module, which consists of five sub-modules:

[0218] (1) Splitting Operation Submodule: When a word in a pictographic language is sent to the word representation module, the splitting operation submodule in the module first splits the input word into characters, and then splits the split characters into individual character shapes (i.e. character shape units) according to the character splitting table.

[0219] (2) Table lookup submodule: This submodule searches pre-stored word lists, character lists, and character shape tables (i.e., character shape unit tables, see diagram). Figure 8 ), obtain the word feature vector (i.e. the first word feature vector) corresponding to the currently input word, and the feature vectors (i.e. the first character feature vector and the character unit feature vector) corresponding to each character and character shape unit obtained by the splitting operation submodule in (1);

[0220] (3) Character-level feature processing submodule (or character-level feature vector processing submodule): First, using the attention-based feature extraction method (the “ABFE” module mentioned above), a fixed-length feature vector is generated from the character-level feature vectors (i.e., character-unit feature vectors) obtained from the character splitting in (1). The specific operation process will be detailed in the next paragraph. This feature vector is called the “extracted character-level feature vector”. The extracted character-level feature vector contains the information most relevant to the meaning of the character, which is the second character feature vector mentioned above. The extracted character-level feature vectors corresponding to each character will be sent to the next submodule.

[0221] The process of generating the extracted word-level feature vectors, such as... Figure 12 As shown. The feature extraction method based on the attention mechanism performs a weighted summation of the glyph feature vectors of all glyphs that make up a single character to obtain the extracted character-level feature vector. The weights of the weighted summation are obtained by calculating the inner product between the glyph feature vectors (glyph unit feature vectors) from the table lookup operation in (2) and the character feature vectors (first character feature vectors). The weights obtained from the inner product calculation are normalized to ensure that the sum of all weights is 1.

[0222] (4) Character-level feature processing submodule (or character-level feature vector processing submodule): This part first fuses the character feature vector from the lookup table operation and the extracted character-level feature vector (second character feature vector) from (3) using a feature fusion method based on gating units (the above "GFF" method) (the specific operation process will be detailed in the next paragraph) to obtain a new character-level feature vector (character representation vector), which is called the "fused character-level feature vector". Then, the feature extraction method based on the attention mechanism is used again to extract a fixed-length feature vector from the fused character-level feature vectors corresponding to each character obtained by splitting (the specific operation process will be detailed in the next paragraph), which is called the "extracted word-level feature vector". The extracted word-level feature vector contains the semantic information most related to the word meaning, that is, the above-mentioned word representation vector. This vector is then sent to the next submodule.

[0223] The process of generating the fused word-level feature vectors is as described above. Figure 15 As shown in the diagram, the feature fusion method based on gating units controls the proportion of extracted character-level feature vectors and character feature vectors in the fusion process through weights (labeled as "gating weights" in the diagram). This weight is a value between 0 and 1, calculated from the character feature vectors through a linear mapping. The proportion of the extracted character-level feature vectors in the fusion process is this weight, and the proportion of the character feature vectors in the fusion process is 1 minus this weight. If the character feature vector from the lookup table operation is valid (i.e., the character is not out of scope and a corresponding character feature vector can be found in the character table), then the character feature vector will have a larger proportion in the fusion process, and correspondingly, the proportion of the extracted character-level feature vector (the second character feature vector) will be relatively small. Conversely, if the character feature vector from the lookup table operation (the first character feature vector) is invalid (i.e., the character is out of scope and a corresponding character feature vector cannot be found in the character table), then the proportion of the extracted character-level feature vector (the second character feature vector) in the fusion process will be very large, and correspondingly, the proportion of the character feature vector (the first character feature vector) will be very small.

[0224] The process of generating the extracted word-level feature vectors is as described above. Figure 13 As shown, the attention-based feature extraction method performs a weighted summation of the fused character-level feature vectors (character representation vectors) corresponding to all individual characters constituting a word, resulting in the extracted word-level feature vector (second word feature vector). The weights of the weighted summation are calculated by the inner product between the fused character-level feature vector (character representation vector) and the word feature vector (first word feature vector) obtained from the table lookup operation. The weights obtained from the inner product calculation are normalized to ensure that the sum of all weights is 1.

[0225] (5) Word-level feature processing submodule (or word-level feature vector submodule or word-level feature vector processing submodule): This submodule uses a feature fusion method based on gating units to fuse the word feature vector (first word feature vector) from the lookup table operation with the word-level feature vector (second word feature vector) extracted from (4) (the specific operation process will be detailed in the next paragraph) to obtain the fused word-level feature vector (word representation vector). This fused word-level feature vector (word representation vector) is used as the final word representation vector generated by the word representation module and output to the next part - the encoder.

[0226] The process of generating the fused word-level feature vectors (word representation vectors), such as... Figure 16 As shown, the feature fusion method based on gated units controls the proportion of extracted word-level feature vectors and word feature vectors in the fusion process through weights (labeled "gating weights" in the figure). These weights are values ​​between 0 and 1, calculated from the word feature vectors (first word feature vectors) through a linear mapping. The proportion of the extracted word-level feature vectors (second word feature vectors) in the fusion is equal to this weight, while the proportion of the word feature vectors in the fusion is 1 minus this weight. If the word feature vector from the lookup operation is valid (i.e., the word is not out-of-scope and a corresponding word feature vector can be found in the word list), then the word feature vector (first word feature vector) will account for a larger proportion in the fusion process, and correspondingly, the proportion of the extracted word-level feature vector (second word feature vector) will be relatively small. If the word feature vector from the lookup operation (first word feature vector) is invalid (i.e., the word is out-of-scope and a corresponding word feature vector cannot be found in the word list), then the proportion of the extracted word-level feature vector (second word feature vector) will be very large in the fusion process, and correspondingly, the proportion of the word feature vector (first word feature vector) will be very small.

[0227] 2. The second part is an encoder, which can be any neural network-based encoder, such as a feedforward neural network encoder, a Long Short-Term Memory (LSTM) network encoder, a Transformer-XL encoder, etc. This encoder encodes the word representation vectors from the multi-level word representation modules to obtain the current context vector, and outputs it to the output layer. (Note: The following examples and "" Figure 17 (The structural diagrams of this invention are all illustrated using an LSTM encoder as an example.)

[0228] 3. The last part is the output layer, which is a single-layer feedforward neural network. Based on the context vector encoded by the encoder, it calculates the probability of each word in the vocabulary appearing in the next position through linear computation and normalization. It then predicts the word in the next position.

[0229] In the above example, a multi-level word representation module HWR is proposed. It considers setting glyph units to create hieroglyphic representations, and designs ABFE and GFF to obtain word representation vectors and character representation vectors. GFF can adjust the weight between the first character feature vector and the second character feature vector, as well as the weight between the first word feature vector and the second word feature vector, according to whether the word itself or the character itself has valid semantic information, so that the finally obtained word representation vector has richer semantic information.

[0230] To better understand the above method for determining word representation vectors, the following details an example of determining word representation vectors according to the present invention:

[0231] In this example, a simplified HWR is used instead of HWR. The simplified HWR only uses glyph units to generate the final word representation vector. The performance of the simplified HWR may not be as good as the original HWR, but it is faster and smaller in size.

[0232] The simplified HWR only retains the compositional information of the input word and directly creates the final word representation (i.e., the word representation vector), including:

[0233] Splitting operation: Split the input character or word into multiple parts, that is, multiple glyph units.

[0234] Lookup table: Assign the feature vectors of the glyph units to each glyph unit obtained by splitting through the lookup table.

[0235] Glyph unit level: ABFE extracts character-level feature vectors from the feature vectors of the glyph units.

[0236] Character level: ABFE extracts word-level feature vectors from these character-level feature vectors.

[0237] Word level: The word-level feature vectors extracted from the character level are the final word representation vectors.

[0238] In an example, as Figure 18 shown, when the current word of the input sentence of the neural network language model is "carry out".

[0239] 1) Split the input "carry out" into "carry" and "out", and respectively split "carry" and "out" to obtain sets of glyph units;

[0240] 2) Perform a lookup operation in the glyph table (i.e., the glyph unit table), query the glyph unit feature vectors corresponding to each glyph unit in the set of glyph units. Taking the "辶" and "井" of the character "进" as an example, query the glyph unit feature vector of "辶", and query the glyph unit feature vector of "井";

[0241] 3) Use ABFE to obtain the glyph unit weight coefficients of "辶" and "井", and based on the glyph unit weight coefficient and glyph unit feature vector of "辶", and the glyph unit weight coefficient and glyph unit feature vector of "井", obtain the second character feature vector of "进". It should be noted that here, there is no need to obtain the first character feature vector of "进" anymore, but directly use the second character feature vector of "进" as the character representation vector;

[0242] 5) Based on the same ABFE method, obtain the second character feature vector of the character "行", and use the second character feature vector of "行" as the character representation vector of "行";

[0243] 6) Use the ABFE method to obtain the character weight coefficients of "进" and "行", and based on the character weight coefficient and character representation vector of "进", and the character weight coefficient and character representation vector of "行", obtain the second word feature vector of "进行";

[0244] 7) Use the second word feature vector of "进行" as the word representation vector of "进行";

[0245] 8) Then input the word representation vector of "进行" into a long short-term memory network (LSTM) for encoding to generate a context vector;

[0246] 9) The output layer calculates the probability that each word in the vocabulary may appear in the next position based on the context vector, and further predicts the word at the next position.

[0247] In the above example, by no longer obtaining the first character feature vector, but directly using the second character feature vector as the character representation vector; and no longer obtaining the first word feature vector, directly using the second word feature vector as the word representation vector, for words or characters with fewer semantic features or no semantic features themselves, the finally obtained word representation vector can not only retain effective semantic features, but also reduce the computational amount and improve the prediction efficiency.

[0248] To better understand the above method for determining the word representation vector, the following details an example of determining the word representation vector of the present invention:

[0249] In one example, as Figure 19 shown, when the current word of the sentence input to the neural network language model is "蒹葭".

[0250] 1. First, this word is sent to the multi-level word representation module and passes through the following several sub-modules in sequence (the specific execution process is as Figure 19 shown):

[0251] (1) Splitting operation sub-module: First, "jiān jiā" is split into two characters, "jiān" and "jiā". According to the character splitting table, the single character "jiān" is further split into two glyph units, "艹" and "jiān". Similarly, according to the character splitting table, "jiā" is split into two glyph units, "艹" and "jiǎ".

[0252] (2) Table lookup operation sub-module: Since "jiān jiā" is an out-of-vocabulary word, the table lookup operation cannot find the corresponding word feature vector. Moreover, the single characters "jiān" and "jiā" that make it up are also out-of-vocabulary characters, and the table lookup operation cannot find their corresponding character feature vectors. Therefore, the feature vectors of the word and character output by the table lookup operation are both invalid. However, the glyph units of the characters can all be found in the glyph unit table (i.e., the glyph table). And the glyph unit "艹" contains valid semantic information - it indicates that the word describes a kind of herbaceous plant. So the table lookup operation sub-module can output valid glyph unit feature vectors (i.e., the valid glyph feature vectors shown in the figure), and these feature vectors contain information related to the word meaning.

[0253] (3) Glyph unit level feature processing sub-module (also called glyph level feature vector processing sub-module): Through the feature extraction method based on the attention mechanism ( Figure 19 the "ABFE" module in it), the extracted character level feature vectors corresponding to these two single characters are generated respectively from the glyph units that make up "jiān" and "jiā". Since "艹" is more relevant to the word meaning, in the weighted summation, the weight of "艹" is the largest. Therefore, the extracted character level feature vectors mainly contain information related to the glyph unit "艹".

[0254] (4) Character level feature processing sub-module (also called character level feature vector processing sub-module): Since both "jiān" and "jiā" are out-of-vocabulary characters, the character feature vectors given by the table lookup operation are both invalid. Therefore, the feature fusion method based on the gated unit ( Figure 19 the "GFF" module in it) will adjust the weights to increase the proportion of the extracted character level feature vectors in the fusion, and the proportion of the character feature vectors will decrease accordingly. This makes the fused character level feature vectors mainly contain the information of the extracted character level feature vectors. In this case, the fused character level feature vectors mainly contain information related to the glyph unit "艹". Subsequently, through the feature extraction method based on the attention mechanism ( Figure 19 the "ABFE" module in it), from the fused character level feature vectors corresponding to the two characters "jiān" and "jiā", the extracted word level feature vector corresponding to "jiān jiā" is generated. Since both characters are quite relevant to the word meaning of "jiān jiā", their weights will be relatively close (the sum of the weights is 1). In this case, the extracted word level feature vector mainly contains information related to the glyph unit "艹".

[0255] (5) Word-level feature processing sub-module (also known as word-level feature vector processing sub-module or word-level feature vector sub-module): Since "jianjia" is an out-of-vocabulary word, the word feature vector obtained by the look-up table operation is invalid. Therefore, the feature fusion method based on the gated unit ( Figure 19 the "GFF" module in) will adjust the weights to increase the proportion of the extracted word-level feature vector in the fusion, and the proportion of the word feature vector will correspondingly decrease. This makes the fused word-level feature vector mainly contain the information of the extracted word-level feature vector. In this case, in the fused word-level feature vector (that is, the word representation vector finally output by the multi-level word representation module), it will mainly contain information related to the glyph unit "艹". The subsequent encoder can still obtain effective semantic information from this word representation vector - that is, the word describes a herbaceous plant.

[0256] 2. Then, the word representation vector generated by the multi-level word representation module is fed into a long short-term memory network (LSTM) for encoding to generate a context vector.

[0257] 3. The output layer uses a feed-forward neural network to calculate the probability that each word in the vocabulary may appear in the next position based on the context vector. Then it predicts the word in the next position.

[0258] In the above example, if the word itself or the character itself does not include or contains less effective semantic information, when obtaining the weighted sum of the first character feature vector and the second character feature vector, or the weighted sum of the first word feature vector and the second word feature vector, the weight of the first character feature vector obtained by query can be reduced, or the weight of the first word feature vector obtained by query can be reduced, so that the finally obtained word representation vector contains more semantic information obtained based on the glyph unit, thereby obtaining more effective semantic information and further improving the accuracy of determining the word representation vector.

[0259] To better understand the above method for determining the word representation vector, the following details an example of determining the word representation vector of the present invention:

[0260] In an example, as Figure 20 shown, taking the processing of "jinxing" as an example, the method for determining the word representation vector provided by this application may include the following steps:

[0261] 1) Split "jinxing" into two characters "jin" and "xing", and respectively split "jin" and "xing" to obtain the glyph unit sets;

[0262] 2) Perform a look-up operation in the glyph table (i.e., the glyph cell table) to query the glyph cell feature vectors corresponding to each glyph cell in the glyph cell set. Taking the "辶" and "井" of the character "进" as an example, query the glyph cell feature vector of "辶" and the glyph cell feature vector of "井".

[0263] 3) Perform a look-up operation in the character table to query the first character feature vector corresponding to "进", and calculate the cosine similarity between the glyph cell feature vector of "辶" and the first character feature vector corresponding to "进"; calculate the cosine similarity between the glyph cell feature vector of "井" and the first character feature vector corresponding to "进".

[0264] 4) Take the obtained cosine similarities as the glyph cell weight coefficients of "辶" and "井" respectively, and obtain the second character feature vector of "进" based on the glyph cell weight coefficient of "辶" and the glyph cell feature vector of "辶", and the glyph cell weight coefficient of "井" and the glyph cell feature vector of "井".

[0265] 5) Obtain the second character feature vector of the character "行" based on the same CosFE method.

[0266] 6) Query the first character feature vector of "进", add the first character feature vector of "进" and the second character feature vector to obtain the character representation vector of "进"; similarly, query the first character feature vector of "行", add the first character feature vector of "行" and the second character feature vector to obtain the character representation vector of "行".

[0267] 7) Perform a look-up operation in the word table to query the first word feature vector of "进行"; calculate the cosine similarity between the character representation vector of "进" and the first word feature vector of "进行"; calculate the cosine similarity between the character representation vector of "行" and the first word feature vector of "进行".

[0268] 8) Take the obtained cosine similarities as the character weight coefficients of "进" and "行" respectively, and obtain the second word feature vector of "进行" based on the character weight coefficient of "进" and the character representation vector of "进", and the character weight coefficient of "行" and the character representation vector of "行".

[0269] 9) Add the first word feature vector of "进行" and the second word feature vector to obtain the word representation vector of "进行".

[0270] 10) Then input the word representation vector of "进行" into a long short-term memory network (LSTM) for encoding to generate a context vector.

[0271] 11) The output layer calculates the probability that each word in the word table may appear in the next position according to the context vector, and further predicts the word in the next position.

[0272] In one example, as Figure 21 shown, taking the processing of "reed" as an example, the method for determining the word representation vector provided in this application may include the following steps:

[0273] 1) Split "reed" into two characters, "芦" and "苇", and respectively split "芦" and "苇" to obtain a set of glyph units. Split "芦" into "艹" and "户", and split "苇" into "艹" and "韦";

[0274] 2) Query the glyph unit feature vectors corresponding to each glyph unit in the set of glyph units. Taking "艹" and "户" of the character "芦" as an example, query the glyph unit feature vector of "艹" and the glyph unit feature vector of "户"; The glyph unit "艹" contains effective semantic information - it indicates that the word describes a kind of herbaceous plant. So the look-up table operation sub-module can output effective glyph unit feature vectors, and these feature vectors contain information related to the word meaning;

[0275] 3) Use ABFE to respectively generate the extracted character-level feature vectors corresponding to these two single characters from the glyph units constituting "芦" and "苇"; Since "艹" is more relevant to the word meaning, in the weighted summation, the weight of "艹" is the largest. Therefore, the extracted character-level feature vectors mainly contain information related to the glyph unit "艹"; The "卢" and "韦" corresponding to the two glyph units shown in the figure are invalid, that is, the semantics of "卢" and "韦" themselves have little relevance to herbaceous plants, that is, they contain less effective semantic information or no effective semantic information at all;

[0276] 4) Based on GFF, adjust the weights to adjust the proportion of the extracted character-level feature vectors and character feature vectors in the fusion. The two characters "芦" and "苇" themselves also contain the meaning of "plant". The fused character-level feature vectors incorporate the information of both the extracted character-level feature vectors and character feature vectors, mainly containing information related to "plant";

[0277] 5) Use ABFE to generate the extracted word-level feature vector corresponding to "reed" from the fused character-level feature vectors corresponding to the two characters "芦" and "苇". The extracted word-level feature vector mainly contains information related to the glyph unit "plant";

[0278] 6) Since the word "reed" itself has the information of "plant", the word feature vector obtained by the look-up table operation is also valid. Based on GFF, adjust the weights so that the fused word-level feature vectors are compatible with the extracted word-level feature vectors and word feature vectors, and effective semantic information can be obtained - that is, the word describes a kind of herbaceous plant;

[0279] 7) The word representation vector generated by the multi-level word representation module generates a context vector; the output layer uses a feedforward neural network to calculate the probability that each word in the vocabulary may appear in the next position based on the context vector, and then predicts the word in the next position.

[0280] In the above example, if the word or character itself has effective semantic information, then when obtaining the weighted sum of the first character feature vector and the second character feature vector, or the weighted sum of the first word feature vector and the second word feature vector, the semantic information contained in the word or character itself can be preserved. Combined with the semantic information in the character unit, the final word representation vector can have richer semantic information, thereby further improving the accuracy of determining the word representation vector.

[0281] The method described above for determining word representation vectors can be applied not only to pictographic scripts, such as Chinese, but also to non-pictographic scripts that can be broken down into basic units such as glyphs or letters, such as Latin. The HWR method described above can also be used for Latin language models, further dividing Latin words into BPE (Byte Pair Encoding) segments. BPE is a compression algorithm that splits words into word fragments. The word level and the word fragment level split by BPE are considered as two levels for words in the HWR method, which then processes Latin words. Compared to charCNN, most features are effective, and this approach can more accurately capture the semantics of words and determine their word representation vectors. The word fragment level split by BPE is similar to the glyph level mentioned above, and the word level is similar to the character level or word level mentioned above.

[0282] For example, in Latin English, such as Figure 22a As shown, firstly, the English word "preprocess" is split into three BPE word fragments: pre, pro, and cess. Then, a lookup table operation is used to retrieve the BPE word fragment feature vector corresponding to the BPE word fragment, and the first word feature vector corresponding to the word is retrieved from the word table. The ABFE method is used to determine the second word feature vector of "preprocess" based on the BPE word fragment feature vectors of pre, pro, and cess. Finally, GFF is used to fuse the first word feature vector and the second word feature vector to obtain the final word representation vector.

[0283] To better understand the application scenarios of the method for determining word representation vectors described above, the following details the application process of the method for determining word representation vectors according to the present invention:

[0284] Taking its application in input methods as an example, such as Figure 22b As shown, methods for determining word representation vectors may include:

[0285] First, the input method UI (User Interface) obtains the user's current input, such as... Figure 23 As shown, for example, the input "Samsung" is sent to the language model; then, the language model uses the method for determining word representation vectors described earlier in this application to process the current input, such as a sequence of words, to obtain the corresponding word representations, encodes them using an encoder, sends them to the output layer, predicts the words that may appear in the next position of the sequence, and sorts them according to their probability, such as ",", "display", "mobile", "of", "period", "pile", and returns the results to the input method UI, which then displays the next word candidate.

[0286] In the process of determining the word representation vector, this example can use ABFE to obtain the second character feature vector and the second word feature vector, and use GFF to adjust the weights between the first character feature vector and the second character feature vector, as well as the weights between the first word feature vector and the second word feature vector, which can improve the accuracy of determining the word representation vector. To verify the effect of the determined word representation vector of this application, Chinese characters evolved from glyphs were tested.

[0287] 1. This application uses more than 6 million statements from online news and 30 books to train the device and uses 20,000 statements to test the device. The test statements are from the same source as the training statements.

[0288] The test results are shown in the table below:

[0289] Table 1

[0290] Confusion Word similarity Existing technology 241.91 31.1208 This application 231.83 50.8317

[0291] Among them: perplexity is a metric used to measure the accuracy of the language model; the lower the perplexity, the more accurate the language model. Word similarity is a metric used to measure the quality of word representation; the higher the word similarity, the more accurate the semantics of the predicted words. 2. Select three commonly used sentences for experiments. Then select the next word prediction scenario for evaluation. This is a basic requirement of input methods and is closely related to user experience. One language model is our new language model, and the other is the original language model. We compare their performance to find out which one is more convenient and faster.

[0292] You can use the following three common sentences to experiment:

[0293] Sentence 1: Let's go take a look together;

[0294] Sentence 2: There is a large building in the distance;

[0295] Sentence 3: Time is of the essence now.

[0296] Table 2

[0297]

[0298] Combine Table 2 and Figure 24 As shown, shorter typing time means that the language model can speed up typing; for each sentence, using the language model of this application, less time is required to type. Compared with existing language models, the language model of this application reduces typing time by a total of 52.63%. In other words, typing becomes faster.

[0299] Table 3

[0300]

[0301] Combined with Table 3 and Figure 25 As shown, the reduced number of clicks means that the language model makes input more convenient; by using the language model of this application, fewer clicks are required to input a sentence. Compared with existing language models, the language model of this application reduces the total number of clicks by 43.75%. In other words, typing becomes more convenient.

[0302] In summary, the language model of this application uses a hierarchical word representation module to create higher-quality word representation vectors, thus enabling it to provide more accurate probabilities for each word in a sentence. Consequently, the input method supported by the language model of this application can provide more accurate predictions of the next word.

[0303] For sentences 1, 2, and 3, the language model of this application predicts the next word that the user will type and provides it directly to the user. Therefore, the user does not need to type each word. Thus, compared to existing language models, the language model of this application makes typing faster and more convenient.

[0304] The actual effect of this application:

[0305] 1. For pictographic languages ​​such as Chinese, language models that use multi-level word representation modules have lower perplexity than other existing language models.

[0306] 2. For words composed of characters outside the syllabus, although neither the word nor the character is in the vocabulary or character list, the word representation vector generated by the word representation module still contains useful semantic information because the language model here introduces information from the glyph units.

[0307] 3. For pictographic languages ​​such as Chinese, the quality of word representation vectors generated by the multi-level word representation module in this language model is higher than that generated by the word representation module of existing methods.

[0308] The above embodiments describe the method for determining word representation vectors from the perspective of the process flow. The following description describes the method from the perspective of virtual modules, as detailed below:

[0309] This application provides an apparatus 2600 for determining word representation vectors, such as... Figure 26 As shown, the device 2600 may include a first acquisition module 2601 and a second acquisition module 2602, wherein:

[0310] The first acquisition module 2601 is used to acquire a set of glyph units of at least one word;

[0311] The second acquisition module 2602 is used to acquire the word representation vector of the at least one word based on the set of glyph units.

[0312] The aforementioned device for determining word representation vectors obtains context vectors from the set of glyph units of the text, and then predicts the next word of the text based on the context vectors. Even for pictographic characters or languages ​​evolved from pictographic characters that are prone to having characters beyond the scope of the text, it can effectively obtain the corresponding set of glyph units, thereby improving the accuracy of determining word representation vectors.

[0313] In one possible implementation of this application embodiment, when the first acquisition module 2601 acquires the set of glyph units of at least one word in the text, it is specifically used for:

[0314] The at least one word is split into at least one character;

[0315] The at least one character obtained from the splitting is further split to obtain at least one glyph unit, and the set of glyph units is obtained based on the at least one glyph unit.

[0316] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the word representation vector of the at least one word based on the set of glyph units, it is specifically used for:

[0317] Obtain the feature vector of the glyph unit in the set of glyph units;

[0318] Based on the feature vector of the character unit, obtain the word representation vector of the at least one word.

[0319] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the word representation vector of the at least one word based on the glyph unit feature vector, it is specifically used for:

[0320] Based on the glyph unit feature vector, obtain the character representation vector corresponding to at least one character in the at least one word;

[0321] Based on the character representation vector corresponding to at least one character in the at least one word, obtain the word representation vector of the at least one word.

[0322] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the character representation vector corresponding to at least one character in the at least one word based on the character shape unit feature vector, it is specifically used for:

[0323] Obtain the first character feature vector of at least one character in the at least one word;

[0324] Based on the glyph unit feature vector, obtain the second character feature vector corresponding to at least one character in the at least one word;

[0325] Based on the first character feature vector and the second character feature vector, obtain the character representation vector corresponding to the at least one character.

[0326] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the first character feature vector of at least one character in the at least one word, it is specifically used for:

[0327] Query the first character feature vector corresponding to at least one character in the at least one word;

[0328] If the corresponding first character feature vector is not found, the preset character feature vector is set as the first character feature vector.

[0329] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the word representation vector of the at least one word based on the character representation vector corresponding to at least one character in the at least one word, it is specifically used for:

[0330] Obtain the first word feature vector of the at least one word;

[0331] Based on the character representation vector corresponding to the at least one character, obtain the second word feature vector of the at least one word;

[0332] Based on the first word feature vector and the second word feature vector, obtain the word representation vector of the at least one word.

[0333] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the first word feature vector of the at least one word, it is specifically used for:

[0334] Query the feature vector of the first word corresponding to the at least one word;

[0335] If the corresponding first word feature vector is not found, the preset word feature vector is set as the first word feature vector.

[0336] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the second character feature vector corresponding to at least one character in the at least one word based on the character unit feature vector, it is specifically used to: determine the character unit weight coefficient corresponding to at least one character in the at least one word;

[0337] Based on the determined character unit weight coefficients and the corresponding character unit feature vectors, the corresponding second character feature vector is obtained.

[0338] In one possible implementation of this application embodiment, when the second acquisition module 2602 determines the weight coefficient of the glyph unit corresponding to at least one character in the at least one word, it is specifically used for:

[0339] The inner product of the glyph feature vector of at least one character's glyph unit and the first character feature vector of the corresponding character is determined to obtain the initial glyph weight of the glyph unit.

[0340] Based on the initial glyph unit weights, the glyph unit weight coefficients of the corresponding glyph units are determined.

[0341] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the character representation vector corresponding to the at least one character based on the first character feature vector and the second character feature vector, it is specifically used for:

[0342] The first character weight coefficient of the first character feature vector is obtained by performing a linear mapping calculation on the first character feature vector;

[0343] The second character weight coefficient is obtained based on the first character weight coefficient;

[0344] The character representation vector of the corresponding character is obtained based on the first character weight coefficient, the first character feature vector, the second character weight coefficient, and the second character feature vector.

[0345] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the second word feature vector of the at least one word based on the character representation vector corresponding to the at least one character, it is specifically used for:

[0346] Determine the inner product of the character representation vector of at least one character in at least one word and the first word feature vector of the corresponding word to obtain the initial character weight of at least one character;

[0347] Based on the initial character weights, obtain the character weight coefficient of at least one character;

[0348] Based on the character weight coefficient of at least one character and the corresponding character representation vector, the second word feature vector of the corresponding word is obtained.

[0349] In one possible implementation of this application embodiment, when the second acquisition module 2602 acquires the word representation vector of the at least one word based on the first word feature vector and the second word feature vector, it is specifically used for:

[0350] The first word weight coefficient of the first word feature vector is obtained by performing a linear mapping on the first word feature vector;

[0351] The second word weight coefficient is obtained based on the first word weight coefficient;

[0352] The word representation vector of the corresponding word is obtained based on the weight coefficient of the first word, the feature vector of the first word, the weight coefficient of the second word, and the feature vector of the second word.

[0353] The apparatus for determining word representation vectors of images according to embodiments of this disclosure can execute a method for determining word representation vectors of images provided in embodiments of this disclosure. The implementation principle is similar. The actions performed by each module in the apparatus for determining word representation vectors of images in various embodiments of this disclosure correspond to the steps in the method for determining word representation vectors of images in various embodiments of this disclosure. For detailed functional descriptions of each module in the apparatus for determining word representation vectors of images, please refer to the descriptions of the corresponding methods for determining word representation vectors of images shown above, which will not be repeated here.

[0354] The above describes the apparatus for determining word representation vectors provided in the embodiments of this application from the perspective of functional modularity. Next, the electronic device provided in the embodiments of this application will be described from the perspective of hardware physicalization, and the computing system of the electronic device will also be described.

[0355] Based on the same principles as the methods shown in the embodiments of this disclosure, the embodiments of this disclosure also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer operation instructions; and the processor for executing the method for determining word representation vectors shown in the embodiments by invoking the computer operation instructions. Compared with the prior art, the method for determining word representation vectors in this application can effectively obtain the corresponding set of glyph units even for pictographic characters or languages ​​evolved from pictographs that are prone to having characters beyond their syllabus, thereby improving the accuracy of determining word representation vectors.

[0356] In one alternative embodiment, an electronic device is provided, such as Figure 27 As shown, Figure 27 The illustrated electronic device 2700 includes a processor 2701 and a memory 2703. The processor 2701 and the memory 2703 are connected, for example, via a bus 2702. Optionally, the electronic device 2700 may also include a transceiver 2704. It should be noted that in practical applications, the transceiver 2704 is not limited to one type, and the structure of this electronic device 2700 does not constitute a limitation on the embodiments of this application.

[0357] Processor 2701 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 2701 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0358] Bus 2702 may include a pathway for transmitting information between the aforementioned components. Bus 2702 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 2702 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 27 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0359] The memory 2703 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0360] The memory 2703 stores application code that executes the scheme of this application, and its execution is controlled by the processor 2701. The processor 2701 executes the application code stored in the memory 2703 to implement the content shown in the foregoing method embodiments.

[0361] Among them, electronic devices include, but are not limited to: mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 27 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0362] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments. Compared with the prior art, the method for determining word representation vectors in this application can effectively obtain the corresponding set of glyph units even for pictographic characters or languages ​​evolved from pictographs that are prone to having out-of-scope characters, thereby improving the accuracy of determining word representation vectors.

[0363] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0364] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0365] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0366] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0367] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0368] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0369] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a module does not necessarily limit the module itself; for example, a prediction module can also be described as "a module for predicting the next word".

[0370] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

Claims

1. A method for determining word representation vectors, characterized in that, include: Obtain at least one word from user input; Input the at least one word into the language model to perform: The at least one word is split into at least one character; By splitting the at least one character into glyph units, a set of glyph units for the at least one word is obtained, wherein the set of glyph units includes at least one glyph unit; Based on the feature vector of the glyph unit of the at least one glyph unit, obtain the character representation vector corresponding to at least one character in the at least one word; Based on the character representation vector corresponding to at least one character in the at least one word, obtain the word representation vector of the at least one word; The process of obtaining the character representation vector corresponding to at least one character in the at least one word includes: Obtain the first character feature vector of at least one character in the at least one word; Based on the feature vector of the glyph unit of the at least one glyph unit, obtain the feature vector of the second character corresponding to at least one character in the at least one word; Based on the first character feature vector and the second character feature vector, obtain the character representation vector corresponding to the at least one character.

2. The method according to claim 1, characterized in that, The step of obtaining at least one word from user input includes: Displays a user interface including a virtual keyboard, suggestion area, and input fields; The virtual keyboard receives at least one word input by the user. The method further includes: Provide predicted words or characters corresponding to the word representation vector; The predicted words or characters are output from the language model; The user interface is controlled to display the predicted word or character in the suggested area of ​​the user interface; Based on the selection of the predicted word or character from the suggestion area, the predicted word or character is displayed in the input field of the user interface.

3. The method according to claim 1, characterized in that, The step of obtaining the word representation vector of the at least one word includes: Based on the word feature vector of the at least one word and the character representation vector of the at least one character, obtain the word representation vector of the at least one word.

4. The method according to claim 1, characterized in that, Obtaining the character representation vector corresponding to at least one character in the at least one word includes: Obtain the glyph feature vector of the at least one glyph unit and the first character feature vector of the at least one character from the lookup table; An attention-based feature extraction method is applied to the feature vector of the at least one glyph unit obtained from the lookup table to obtain the second character feature vector corresponding to the at least one character; The first character feature vector obtained from the lookup table is fused with the extracted second character feature vector to obtain the character representation vector corresponding to at least one character in the at least one word.

5. The method according to claim 4, characterized in that, The lookup table includes a character table, and obtaining the first character feature vector of at least one character in the at least one word includes: Search the character table for the first character feature vector corresponding to at least one character in the at least one word; If the first character feature vector corresponding to the at least one character is not found in the character table, the preset character feature vector is set as the first character feature vector.

6. The method according to claim 1, characterized in that, Obtaining the word representation vector of the at least one word based on the character representation vector corresponding to at least one character in the at least one word includes: Obtain the first word feature vector of the at least one word; Based on the character representation vector corresponding to the at least one character, obtain the second word feature vector of the at least one word; Based on the first word feature vector and the second word feature vector, obtain the word representation vector of the at least one word.

7. The method according to claim 6, characterized in that, The step of obtaining the first word feature vector of the at least one word includes: Search the word list for the first word feature vector corresponding to the at least one word; If the first word feature vector corresponding to the at least one word is not found in the word list, the preset word feature vector is set as the first word feature vector.

8. The method according to claim 1, characterized in that, Based on the feature vector of the at least one glyph unit, obtain the second character feature vector corresponding to at least one character in the at least one word, including: Determine the weight coefficient of the glyph unit corresponding to at least one character in the at least one word; Based on the glyph unit feature vector and the weight coefficient of the glyph unit corresponding to at least one character in the at least one word, the corresponding second character feature vector is obtained.

9. The method according to claim 8, characterized in that, Determining the weight coefficients of the glyph units corresponding to at least one character in the at least one word includes: The inner product of the glyph feature vector of the character's glyph unit and the first character feature vector of the character is determined to obtain the initial glyph unit weight of the character's glyph unit; Based on the initial glyph unit weights, the glyph unit weight coefficients of the corresponding glyph units are determined.

10. The method according to claim 1, 5, or 8, characterized in that, The step of obtaining the character representation vector corresponding to the at least one character based on the first character feature vector and the second character feature vector includes: The first character weight coefficient of the first character feature vector is obtained by performing a linear mapping calculation on the first character feature vector; Based on the weight coefficient of the first character, obtain the weight coefficient of the second character in the feature vector of the second character; The character representation vector of the corresponding character is obtained based on the first character weight coefficient, the first character feature vector, the second character weight coefficient, and the second character feature vector.

11. The method according to claim 6, characterized in that, The step of obtaining the second word feature vector of the at least one word based on the character representation vector corresponding to the at least one character includes: The initial character weight of at least one character is obtained by determining the inner product of the character representation vector of at least one character in at least one word and the first word feature vector of the corresponding word. Based on the initial character weights, obtain the character weight coefficient of at least one character; Based on the character weight coefficient of at least one character and the corresponding character representation vector, the second word feature vector of the corresponding word is obtained.

12. The method according to claim 6, 7, or 11, characterized in that, The step of obtaining the word representation vector of the at least one word based on the first word feature vector and the second word feature vector includes: The first word weight coefficient of the first word feature vector is obtained by performing a linear mapping on the first word feature vector; The second word weight coefficient is obtained based on the first word weight coefficient; The word representation vector of the corresponding word is obtained based on the weight coefficient of the first word, the feature vector of the first word, the weight coefficient of the second word, and the feature vector of the second word.

13. The method according to any one of claims 1 to 9, characterized in that, The step of obtaining the word representation vector of the at least one word includes: Using a lookup table, obtain the first word feature vector of the at least one word, the first character feature vector of the at least one character, and the character unit feature vector of the at least one character unit; An attention-based feature extraction method is applied to the feature vector of the character unit to obtain the second character feature vector; By combining the first character feature vector obtained using the lookup table and the extracted second character feature vector, a character representation vector is obtained. An attention-based feature extraction method is applied to the word representation vector to obtain the extracted second word feature vector; The word representation vector is obtained by fusing the extracted second word feature vector with the first word feature vector obtained using a lookup table.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory: The memory is used to store machine-readable instructions; The processor is configured to execute the method of any one of claims 1 to 13 when executing the machine-readable instructions.

15. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions, which, when executed by a processor, implement the method of any one of claims 1 to 13.

Images