A language processing-based word vector representation method, device and terminal equipment

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a word vector representation method and device based on language processing and a terminal device, the method comprising: constructing an input word table; setting word frequency information of each word in the input word table; classifying the words into N clusters according to the word frequency information, words with the same word frequency information being classified into the same set to obtain a clustering set with N subsets; defining a first word vector dimension of the words in the first cluster through an experience factor, and obtaining a second word vector dimension of the words in each subset according to the subset where the words are located and the first word vector dimension; defining an nth linear mapping matrix for the nth cluster; and processing a target word vector through parameters shared by the input word table as an input of a natural language processing model. Through the application, the problem that the calculation amount is large, the efficiency is low and the word distribution features cannot be considered when converting words into word vectors in natural language processing can be solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a word vector representation method, apparatus and terminal device based on language processing. Background Technology

[0002] Compared to images or speech, text is more complex in natural language processing. For example, for neural network-based NLP tasks, the model cannot directly use words as input features, so words must be converted into word vectors.

[0003] A common way to convert words into word vectors is to transform them from one-hot representations into lower-dimensional word vectors through a fully connected layer to reduce the computational cost and parameter size of the model. In the final output stage of the model, it is then transformed into a probability output vector with the same dimension as the vocabulary size through another fully connected layer.

[0004] However, in practical applications, the computational load of the model increases as the vocabulary size grows. Furthermore, frequently used words often account for less than 20% of the total vocabulary, yet their usage rate often exceeds 80%. The word vector conversion described above, applied to the entire vocabulary, not only increases redundant computation but also ignores vocabulary distribution characteristics, affecting the model's final accuracy. Summary of the Invention

[0005] The main objective of this invention is to propose a word vector representation method, apparatus, and terminal device based on language processing. Through a new word vector representation method, it solves the problems of high computational cost, low efficiency, and inability to consider word distribution features in natural language processing when converting words into word vectors.

[0006] To achieve the above objectives, a first aspect of the present invention provides a word vector representation method based on language processing, characterized in that it includes:

[0007] Construct the input vocabulary;

[0008] Set the word frequency information for each word in the input vocabulary;

[0009] Based on the word frequency information, the vocabulary is classified into N clusters. Vocabulary with the same word frequency information is classified into the same set, resulting in a cluster set with N subsets.

[0010] The first word vector dimension of the vocabulary in the first cluster is defined by empirical factors, and the second word vector dimension of the vocabulary in each subset is obtained according to the subset to which the vocabulary belongs and the first word vector dimension.

[0011] For the nth cluster, define the nth linear mapping matrix;

[0012] Obtain the target word vector, wherein the target word vector and the input vocabulary are a shared vocabulary;

[0013] Obtain the corresponding linear mapping matrix by the cluster to which the target word vector belongs, and use the corresponding linear mapping matrix to upsample the word vector of the target word vector to update the target word vector;

[0014] The updated target word vectors are used as input to the natural language processing model;

[0015] Where N is a positive integer, and n is a positive integer less than or equal to N.

[0016] In conjunction with the first aspect of the present invention, in the first embodiment of the present invention, a first word vector dimension of the vocabulary in the first cluster is defined by an empirical factor; and a second word vector dimension of the vocabulary in each subset is obtained based on the subset to which the vocabulary belongs and the first word vector dimension, including:

[0017] The first word vector dimension of the words in the first cluster is defined as d by the empirical factor k. When the subset in which the word belongs is the first subset, the second word vector dimension of the words in the first subset is d.

[0018] When the subset containing the vocabulary is the Nth subset, the second word vector dimension of the vocabulary in the Nth subset is d / k. N-1 ;

[0019] Where k is a natural number greater than 0 and less than 1.

[0020] In conjunction with the first aspect of the present invention, in the second embodiment of the present invention, obtaining the corresponding linear mapping matrix through the cluster to which the target word vector belongs includes:

[0021] A multivariate classifier is used to analyze the cluster to which each word in the target word vector belongs.

[0022] In conjunction with the first aspect of the present invention, in the third embodiment of the present invention, after using the updated target word vector as input to the natural language processing model, the method includes:

[0023] The dimensionality of the word vectors in the updated cluster set is reduced by using the transpose of the corresponding linear vector matrix.

[0024] In conjunction with the first aspect of the present invention, in the fourth embodiment of the present invention, the expression formula for the linear mapping matrix is ​​as follows:

[0025]

[0026] In conjunction with the first aspect of the present invention, in the fifth embodiment of the present invention, each word in the input vocabulary has a different word vector representation.

[0027] In conjunction with the first aspect of the present invention, in the sixth embodiment of the present invention, the word frequency information of the words in the n clusters is less than the word frequency information of the words in the (n+1)th cluster; the word frequency information of the words in the n subsets is less than the word frequency information of the words in the (n+1)th subset.

[0028] A second aspect of the present invention provides a word vector representation device based on language processing, comprising:

[0029] The input vocabulary building module is used to build the input vocabulary.

[0030] The word frequency information input module is used to set the word frequency information for each word in the input word list;

[0031] The vocabulary classification module is used to classify the vocabulary into N clusters based on the word frequency information. Vocabulary with the same word frequency information is classified into the same set to obtain a cluster set with N subsets.

[0032] The word vector dimension calculation module is used to define the first word vector dimension of the words in the first cluster through empirical factors, and to obtain the second word vector dimension of the words in each subset based on the subset to which the words belong and the first word vector dimension.

[0033] The linear mapping matrix setting module is used to define the nth linear mapping matrix for the nth cluster;

[0034] The target word vector acquisition module is used to acquire target word vectors, wherein the target word vectors and the input word list are a shared word list;

[0035] The cluster set update module is used to obtain the corresponding linear mapping matrix through the cluster to which the target word vector belongs, and use the corresponding linear mapping matrix to upsample the word vector of the target word vector to update the target word vector;

[0036] The word vector representation output module is used to take the updated target word vectors as input to the natural language processing model;

[0037] Where N is a positive integer, and n is a positive integer less than or equal to N.

[0038] The word vector representation method based on language processing provided in this invention generates word vectors of different dimensions for words with different word frequencies, and then represents them with different word vector representations. This allows words with higher word frequencies to have more dimensions for feature representation, while words with lower word frequencies have fewer dimensions for feature representation, thereby reducing the number of model parameters and reducing the learning difficulty of the model. Attached Figure Description

[0039] Figure 1A schematic diagram illustrating the implementation process of the word vector representation method based on language processing provided in this embodiment of the invention;

[0040] Figure 2 A schematic diagram of the composition structure of a word vector representation device based on language processing provided in an embodiment of the present invention.

[0041] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0042] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0043] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0044] In this document, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and do not have any specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0045] like Figure 1 As shown, embodiments of the present invention provide a word vector representation method based on language processing, including but not limited to the following steps:

[0046] S100, Construct the input vocabulary.

[0047] In step S100 above, the input vocabulary is a pre-set vocabulary database, and each word in the input vocabulary has a different word vector representation. For example, if the constructed dictionary is: {“John”:1,“likes”:2,“to”:3,“watch”:4,“movies”:5,“also”:6,“football”:7,“games”:8,“Mary”:9,“too”:10}, then the word vector representation is: John:[1,0,0,0,0,0,0,0,0,0].

[0048] The above-mentioned vocabulary list is a conventional vocabulary list, which can represent less information. In this embodiment of the invention, the vocabulary in the input vocabulary list is further processed through the following steps S101 to S104.

[0049] S101. Set the word frequency information for each word in the input word list.

[0050] In step S101 above, the frequency of occurrence of words is word frequency information.

[0051] S102. Classify the vocabulary into N clusters based on the word frequency information. Vocabulary with the same word frequency information is classified into the same set to obtain a cluster set with N subsets.

[0052] In step S102 above, after classifying into N clusters, if the word frequency information of words A and B is the same, then words A and B belong to the same cluster.

[0053] In step S102 above, in a subset, all words have the same word frequency information. Preferably, the word frequency information of the words in the first subset belongs to the word frequency information represented by the first cluster, the word frequency information of the words in the second subset belongs to the word frequency information represented by the second cluster, and the word frequency information of the words in the nth subset belongs to the word frequency information represented by the nth cluster. In this embodiment of the invention, the word frequency information of the words in the n clusters is less than the word frequency information of the words in the (n+1)th cluster. The word frequency information of the words in the n subsets is less than the word frequency information of the words in the (n+1)th subset, that is, the word frequency information represented by the first cluster to the Nth cluster is sorted according to frequency 1 to frequency N.

[0054] In this embodiment of the invention, words in the input vocabulary are clustered based on word frequency information, which greatly reduces the number of parameters and computational complexity in word vector generation during the word vector representation stage.

[0055] S103. Define the first word vector dimension of the words in the first cluster by using empirical factors, and obtain the second word vector dimension of the words in each subset according to the subset where the words are located and the first word vector dimension.

[0056] It should be noted that, in this embodiment of the invention, the second word vector dimension of the vocabulary in each subset is also used as the word vector representation of the vocabulary in each subset. In this case, the word vector representation includes its probability output vector of the same dimension in the input vocabulary.

[0057] In step S103 above, the first word vector dimension of the vocabulary in the first cluster is defined by empirical factors; the second word vector dimension of the vocabulary in each subset is obtained based on the subset to which the vocabulary belongs and the first word vector dimension, including:

[0058] The first word vector dimension of the words in the first cluster is defined as d by the empirical factor k. When the subset in which the word belongs is the first subset, the second word vector dimension of the words in the first subset is d.

[0059] When the subset where the word is located is the Nth subset, the second word vector dimension of the words in the Nth subset is d / k N-1 .

[0060] Therefore, in the embodiments of the present invention, first, the first word vector dimension in the first cluster is defined by an empirical factor, and the words in the same cluster have the same first word vector dimension. The words in the same subset belong to the same cluster. Furthermore, the word vector dimension of the words in the subset can be obtained by calculating based on the first word vector dimension and the subset to which it belongs.

[0061] For example, in the input vocabulary, there are words "bring", "where", "come back", "have a meal", and the word frequency information of "bring" is 2, the word frequency information of "where" is 3, the word frequency information of "come back" is 5, and the word frequency information of "have a meal" is 3. Then, according to the above step S102, the word frequency information is divided into three clusters, specifically: cluster a - the word frequency information is 2, cluster b - the word frequency information is 3, cluster c - the word frequency information is 5. Then, according to the above step S103, the words with the same word frequency information are classified into the same set, and a clustered set with 3 subsets is obtained, specifically: subset a: {bring}, subset b: {where, have a meal}, subset c: {come back}. The first word vector dimension of cluster a is d, the second word vector dimension of the word {bring} is d, and the second word vector dimension of {come back} is d / k 2 . For using the second word vector dimension of the words in each subset as the word vector representation of the words in each subset, then, by way of example, the word vector representation of the word {bring} is d.

[0062] S104. Define the nth linear mapping matrix for the nth cluster.

[0063] In the above step S104, the linear mapping matrix is used to upsample the word vectors of the words in each cluster, so that when the words are input into the natural language processing model, they have the same dimension. Therefore, although the dimension increases during upsampling, it can reduce the complexity of the vector representation.

[0064] It should be noted that for the clusters with a higher ranking, the words in them belong to low-frequency words. After upsampling, the learning difficulty of the natural language processing model for these words is significantly reduced, increasing the learning ability of the natural language processing model and making the perplexity of the final result of the natural language processing task learned lower.

[0065] S105. Obtain the target word vector, and the target word vector and the input vocabulary are a shared vocabulary.

[0066] In the above step S105, the shared vocabulary means that the target word vector can use the parameters in the input vocabulary, such as word frequency information, cluster information to which it belongs, subset information to which it belongs, word vector representation, word vector dimension, etc., which can further reduce the parameter complexity of the natural language processing model.

[0067] S106. Obtain the corresponding linear mapping matrix through the cluster to which the target word vector belongs, and use the corresponding linear mapping matrix to upsample the word vector of the target word vector to update the target word vector.

[0068] In this case, the updated target word vectors all have the same word vector dimension.

[0069] In steps S105 and S106 above, the target word vector is the set of words that the user needs to input into the natural language processing model. The words included are contained in the input vocabulary list pre-constructed in this embodiment of the invention. Therefore, the words in the target word vector have word frequency information, cluster information, subset information, and word vector representation.

[0070] For each word, upsampling is performed using the corresponding linear mapping matrix. After sampling, the words have the same dimension when input into the natural language processing model. The above steps update the target word vector, specifically by updating the word vector dimension.

[0071] The step of obtaining the corresponding linear mapping matrix by the cluster to which the target word vector belongs includes: using a multivariate classifier to analyze the cluster to which each word in the target word vector belongs.

[0072] S107. Use the updated target word vectors as input to the natural language processing model.

[0073] It should be noted that the updated target word vectors still retain their word vector representation in the target vocabulary. In this embodiment of the invention, the word vector representation is the probability output vector of the same dimension as that in the input vocabulary. After inputting into the natural language processing model, all word vectors have the same dimension.

[0074] In practical applications, after step S107 above, the natural language processing model will provide a fully connected layer, and then perform the calculation of the conventional activation function. If the natural language processing model is a translation model, it will obtain the translation prediction word of the target word vector.

[0075] In one embodiment, after step S107 above, the method further includes:

[0076] The dimensionality of the word vectors in the updated cluster set is reduced by using the transpose of the corresponding linear vector matrix.

[0077] In this embodiment of the invention, the dimensionality reduction process described above transforms the word vector dimension of the vocabulary into a lower dimension, thereby reducing the computational load and the number of parameters in the natural language processing model.

[0078] The formula for expressing the linear mapping matrix used in steps S104 and S106 above is as follows:

[0079]

[0080] In this embodiment of the invention, N is a positive integer, n is a positive integer less than or equal to N, and k is a natural number greater than 0 and less than 1.

[0081] The word vector representation method based on language processing provided in this invention generates word vectors of different dimensions for words with different word frequencies, and then represents them with different word vector representations. This allows words with higher word frequencies to have more dimensions for feature representation, while words with lower word frequencies have fewer dimensions for feature representation, thereby reducing the number of model parameters and the learning difficulty of the model. This solves the problems in natural language processing where converting words into word vectors is computationally intensive, inefficient, and cannot consider word distribution features.

[0082] like Figure 2 As shown, this embodiment of the invention also provides a word vector representation device 20 based on language processing, comprising:

[0083] Input vocabulary construction module 200 is used to construct the input vocabulary;

[0084] The word frequency information input module 201 is used to set the word frequency information of each word in the input word list;

[0085] The vocabulary classification module 202 is used to classify the vocabulary into N clusters according to the word frequency information, and to classify the vocabulary with the same word frequency information into the same set to obtain a cluster set with N subsets;

[0086] The word vector dimension calculation module 203 is used to define the first word vector dimension of the words in the first cluster through empirical factors, and to obtain the second word vector dimension of the words in each subset according to the subset to which the words belong and the first word vector dimension;

[0087] Linear mapping matrix setting module 204 is used to define the nth linear mapping matrix for the nth cluster;

[0088] The target word vector acquisition module 205 is used to acquire target word vectors, wherein the target word vectors and the input word list are a shared word list;

[0089] Cluster set update module 206 is used to obtain the corresponding linear mapping matrix through the cluster to which the target word vector belongs, and use the corresponding linear mapping matrix to upsample the word vector of the target word vector to update the target word vector;

[0090] The word vector representation output module 207 is used to take the updated target word vector as input to the natural language processing model;

[0091] Where N is a positive integer, and n is a positive integer less than or equal to N.

[0092] This invention also provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements each step of the word vector representation method based on language processing as described in the above embodiments.

[0093] This invention also provides a storage medium, which is a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the various steps of the word vector representation method based on language processing as described in the above embodiments.

[0094] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the foregoing embodiments have described the present invention in detail, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A word vector representation method based on language processing, characterized in that, include: Construct the input vocabulary; Set the word frequency information for each word in the input vocabulary; Based on the word frequency information, the vocabulary is classified into N clusters. Vocabulary with the same word frequency information is classified into the same set, resulting in a cluster set with N subsets. The first word vector dimension of the vocabulary in the first cluster is defined by empirical factors, and the second word vector dimension of the vocabulary in each subset is obtained according to the subset to which the vocabulary belongs and the first word vector dimension. For the nth cluster, define the nth linear mapping matrix; Obtain the target word vector, wherein the target word vector and the input vocabulary are a shared vocabulary; Obtain the corresponding linear mapping matrix by the cluster to which the target word vector belongs, and use the corresponding linear mapping matrix to upsample the word vector of the target word vector to update the target word vector; The updated target word vectors are used as input to the natural language processing model; Where N is a positive integer, and n is a positive integer less than or equal to N; The step of obtaining the second word vector dimension of the words in each subset based on the subset where the words belong and the first word vector dimension includes: The first word vector dimension of the words in the first cluster is defined as d by the empirical factor k. When the subset in which the word belongs is the first subset, the second word vector dimension of the words in the first subset is d. When the subset containing the vocabulary is the Nth subset, the second word vector dimension of the vocabulary in the Nth subset is: ; Where k is a natural number greater than 0 and less than 1.

2. The word vector representation method based on language processing as described in claim 1, characterized in that, Obtain the corresponding linear mapping matrix by the cluster to which the target word vector belongs, including: A multivariate classifier is used to analyze the cluster to which each word in the target word vector belongs.

3. The word vector representation method based on language processing as described in claim 1, characterized in that, After using the updated target word vectors as input to the natural language processing model, the following steps are taken: The dimensionality of word vectors in the updated cluster set is reduced by using the transpose of the corresponding linear vector matrix.

4. The word vector representation method based on language processing as described in claim 1, characterized in that, The formula for expressing a linear mapping matrix is: 。 5. The word vector representation method based on language processing as described in claim 1, characterized in that, Each word in the input vocabulary has a different word vector representation.

6. The word vector representation method based on language processing as described in claim 1, characterized in that, The word frequency information of words in n clusters is less than that of words in the (n+1)th cluster; the word frequency information of words in n subsets is less than that of words in the (n+1)th subset.

7. A word vector representation device based on language processing, characterized in that, include: The input vocabulary building module is used to build the input vocabulary. The word frequency information input module is used to set the word frequency information for each word in the input word list; The vocabulary classification module is used to classify the vocabulary into N clusters based on the word frequency information. Vocabulary with the same word frequency information is classified into the same set to obtain a cluster set with N subsets. The word vector dimension calculation module is used to define the first word vector dimension of the words in the first cluster through empirical factors, and to obtain the second word vector dimension of the words in each subset based on the subset to which the words belong and the first word vector dimension. The linear mapping matrix setting module is used to define the nth linear mapping matrix for the nth cluster; The target word vector acquisition module is used to acquire target word vectors, wherein the target word vectors and the input word list are a shared word list; The cluster set update module is used to obtain the corresponding linear mapping matrix through the cluster to which the target word vector belongs, and use the corresponding linear mapping matrix to upsample the word vector of the target word vector to update the target word vector; The word vector representation output module is used to take the updated target word vectors as input to the natural language processing model; Where N is a positive integer, and n is a positive integer less than or equal to N; The step of obtaining the second word vector dimension of the words in each subset based on the subset where the words belong and the first word vector dimension includes: The first word vector dimension of the words in the first cluster is defined as d by the empirical factor k. When the subset in which the word belongs is the first subset, the second word vector dimension of the words in the first subset is d. When the subset containing the vocabulary is the Nth subset, the second word vector dimension of the vocabulary in the Nth subset is: ; Where k is a natural number greater than 0 and less than 1.

8. A terminal device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements each step of the word vector representation method based on language processing as described in any one of claims 1 to 6.

9. A storage medium, said storage medium being a computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements each step of the word vector representation method based on language processing as described in any one of claims 1 to 6.

Images