A word vector compression method and device based on frequency domain transformation

Through Fourier transform and frequency domain sorting technology, the problems of high maintenance costs and insufficient expression capabilities in word vector compression are solved, effective compression of highly nonlinear languages ​​and retention of key information are achieved, and the maintenance cost of word vectors is reduced .

CN113642293BActive Publication Date: 2024-11-22HANGZHOU NETEASE ZHIQI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110898531.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-05
Publication Date
2024-11-22
Estimated Expiration
2041-08-05

AI Technical Summary

Technical Problem

In the prior art, it is difficult to effectively reduce maintenance costs when compressing word vectors, while maintaining vocabulary and expression capabilities. Especially when dealing with highly nonlinear distribution languages, clustering and principal component analysis methods have accuracy and linear dimensionality reduction. limitation.

Method used

The word vector is converted into a frequency domain vector through Fourier transform, and sorts based on the modulus size of the frequency domain vector elements, and selects key elements to construct the compressed word vector to achieve effective compression of the word vector.

Benefits of technology

Without reducing vocabulary, highly nonlinear distribution languages ​​can be processed, unimportant elements are removed, and only key information is retained, thereby reducing the maintenance cost of word vectors and improving computing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113642293B_ABST
    Figure CN113642293B_ABST
Patent Text Reader

Abstract

The embodiments of the present disclosure provide a word vector compression method based on frequency domain transformation. The method includes: performing Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector; calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical value of the modulus; selecting a number of elements from the sorted frequency domain vector, and constructing a compressed word vector based on the selected elements. Through the above technical solution, the original expression ability of the word vector is not changed, and the importance of each element of the word vector can be sorted. Therefore, it can not only process vocabulary in a language with a highly nonlinear distribution, but also remove unimportant elements and retain only elements representing key information in the word vector, thereby achieving compression of the word vector and reducing the maintenance cost of the word vector.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology. More specifically, the embodiments of the present disclosure relate to a word vector compression method, device, storage medium and electronic device based on frequency domain transformation. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Word vector refers to the use of vector form in mathematics to represent the vocabulary in a language, that is, mapping the vocabulary into the vector space and representing it with a vector.

[0004] For example, “apple” can be represented as [0.343434,-0.88749,0.992112…,0.232432].

[0005] The elements in the above word vector can be used to indicate in which sentences "apple" appears, which words often come before "apple" and which words often come after "apple" when it appears, where "apple" usually appears in the sentence, and the length of the sentence in which "apple" appears.

[0006] Usually, methods such as word2vec and glove can be used to obtain word vectors based on the massive corpus of tens of billions of words in the corpus through neural network model training to represent the information of a certain word in the corpus.

[0007] In order to improve the representation effect of word vectors, we can increase the dimension of word vectors to represent more information. A corpus of tens of billions of words usually contains a lot of information. Therefore, in order to include this information, the length of the word vector needs to be as long as possible.

[0008] In practical applications, the total vocabulary is often very large. For example, the vocabulary of Chinese is about 500,000. Assuming that the length of the word vector corresponding to each word is 200, a total of about 100 million vector elements are required to represent all Chinese words. This will inevitably lead to the problem of word vectors occupying too much storage space. At the same time, it will also lead to the problem of excessive computational effort when processing word vectors, ultimately increasing the maintenance cost of word vectors. Summary of the invention

[0009] Therefore, there is a great need for an improved word vector compression scheme that can accurately compress highly nonlinearly distributed human language vocabulary without reducing the vocabulary size, thereby reducing the cost of word vector maintenance.

[0010] In this context, the embodiments of the present disclosure are intended to provide a word vector compression method and device based on frequency domain transformation.

[0011] In a first aspect of the embodiments of the present disclosure, a word vector compression method based on frequency domain transformation is provided, comprising:

[0012] Performing Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector;

[0013] Calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical values ​​of the modulus;

[0014] A number of elements are selected from the sorted frequency domain vector, and a compressed word vector is constructed based on the selected elements.

[0015] In one embodiment of the present disclosure, the Fourier transform algorithm used to perform Fourier transform on the word vector to be compressed includes discrete Fourier transform;

[0016] The Fourier transforming of the word vector to be compressed includes:

[0017] The word vector to be compressed is subjected to discrete Fourier transform based on the following formula:

[0018]

[0019] Among them, [X0, X1, …, x N-1 ] T represents the word vector to be compressed, [Y0, Y1, ..., Y N-1 ] T represents the frequency domain vector, Y k =a k +b k i,i 2 =-1, 0≤k≤N-1, W N represents a factor matrix of size N×N,

[0020]

[0021] Among them, ω N Represents the factor, ω N =e^(-2πi / N).

[0022] In one embodiment of the present disclosure, the Fourier transform algorithm used to perform Fourier transform on the word vector to be compressed includes fast Fourier transform;

[0023] The Fourier transforming of the word vector to be compressed includes:

[0024] The word vector to be compressed is subjected to a fast Fourier transform based on the following formula:

[0025]

[0026] in, Indicates size The identity matrix of Indicates size The diagonal matrix of

[0027]

[0028] Indicates size The factor matrix of

[0029]

[0030] in, Represents the factor,

[0031] In one embodiment of the present disclosure, the selecting a number of elements from the sorted frequency domain vector includes:

[0032] When the sorting strategy is to sort the values ​​of the module values ​​from large to small, select the first several elements in the sorting result.

[0033] In one embodiment of the present disclosure, the method further includes:

[0034] Obtain position information corresponding to the plurality of elements in the frequency domain vector before sorting.

[0035] In one embodiment of the present disclosure, constructing a compressed word vector based on the selected elements includes:

[0036] Taking the position information corresponding to the plurality of elements and the modulus values ​​of the plurality of elements as new elements respectively, and constructing the compressed word vector based on the sorting of the plurality of elements;

[0037] or,

[0038] Taking the position information corresponding to the plurality of elements and the square of the modulus values ​​of the plurality of elements as new elements respectively, and constructing the compressed word vector based on the sorting of the plurality of elements;

[0039] or,

[0040] The position information corresponding to the elements, the real parts of the elements, and the imaginary parts of the elements are respectively used as new elements, and the compressed word vector is constructed based on the sorting of the elements.

[0041] In one embodiment of the present disclosure, the position information includes the inverse of position information corresponding to the plurality of elements in the frequency domain vector before sorting.

[0042] In a second aspect of the embodiments of the present disclosure, there is provided an apparatus comprising:

[0043] A transformation module performs Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector;

[0044] A sorting module, calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical values ​​of the modulus;

[0045] A construction module selects a number of elements from the sorted frequency domain vector, and constructs a compressed word vector based on the selected elements.

[0046] In a third aspect of the embodiments of the present disclosure, a medium is provided; a computer instruction is stored on the medium, and when the instruction is executed by a processor, the steps of the method described below are implemented:

[0047] Performing Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector;

[0048] Calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical values ​​of the modulus;

[0049] A number of elements are selected from the sorted frequency domain vector, and a compressed word vector is constructed based on the selected elements.

[0050] In a fourth aspect of the embodiments of the present disclosure, there is provided a computing device, comprising:

[0051] A processor; and a memory for storing instructions executable by the processor;

[0052] The processor executes the executable instructions to implement the following steps:

[0053] Performing Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector;

[0054] Calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical values ​​of the modulus;

[0055] A number of elements are selected from the sorted frequency domain vector, and a compressed word vector is constructed based on the selected elements.

[0056] The above embodiments of the present disclosure have at least the following beneficial effects:

[0057] The word vector regarded as a signal wave is transformed into the frequency domain space through Fourier transform to obtain the frequency domain vector of the same length, and then the modulus of each element of the frequency domain vector is calculated, and sorted based on the size of the modulus value, so as to select several elements to construct the compressed word vector. Through the above technical scheme, on the one hand, since the vocabulary is not reduced when compressing the word vector, the original expression ability of the word vector is not changed; on the other hand, since the word vector is transformed into the frequency domain space, it can be sorted based on the size of the modulus value of each element in the frequency domain vector, and the importance of each element of the word vector can be sorted. Therefore, it can not only process the vocabulary in the language with highly nonlinear distribution, but also remove unimportant elements, and only retain the elements representing the key information in the word vector, so as to achieve the compression of the word vector and reduce the maintenance cost of the word vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0059] Figure 1 A flowchart of a word vector compression method based on frequency domain transformation according to an embodiment of the present disclosure is schematically shown;

[0060] Figure 2 A block diagram of a word vector compression device based on frequency domain transformation according to an embodiment of the present disclosure is schematically shown;

[0061] Figure 3 A schematic diagram of a word vector compression medium based on frequency domain transformation according to an embodiment of the present disclosure is schematically shown;

[0062] Figure 4 A schematic diagram of an electronic device capable of implementing the above method according to an embodiment of the present disclosure is schematically shown.

[0063] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0064] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0065] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0066] According to an embodiment of the present disclosure, a word vector compression method, medium, apparatus and computing device based on frequency domain transformation are proposed.

[0067] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0068] At present, the word vector matrix compression method based on clustering and principal component analysis is usually used to reduce the number of rows and columns of the word vector matrix respectively to compress the word vector, thereby reducing the maintenance cost of the word vector.

[0069] For example, for word vectors to be compressed, the word vectors can be clustered first to remove semantically similar words, thereby reducing the number of rows in the word vector matrix. Then, principal component analysis can be performed on the features of the word vectors to retain only the main features in the word vectors, thereby reducing the number of columns in the word vector matrix and achieving compression of the word vectors.

[0070] However, on the one hand, when performing row compression of word vectors, the clustering results may not be accurate due to the richness and diversity of the vocabulary in the language, and the total amount of vocabulary after clustering is reduced, resulting in limited expressive power of the compressed vocabulary. On the other hand, when performing column compression of word vectors, compression based on principal component analysis is essentially a linear dimensionality reduction technology, which is usually suitable for situations where the spatial distribution is Gaussian, while the distribution of vocabulary in actual language is usually highly nonlinear, resulting in the results obtained based on principal component analysis not necessarily being accurate.

[0071] Next, we will use a simple example to illustrate the compression process of the above word vector. Please refer to Table 1 below. Assume that the vocabulary size is 4 and the length of each word vector is 10.

[0072] The word vectors shown in Table 1 are compressed.

[0073] vocabulary Word vectors Happy [0.8966,0.8534,…,0.9001] joy [0.7902,0.9432,…,0.8349] sad [0.1731,0.2419,…,0.3453] sad [0.2285,0.2943,…,0.3717]

[0074] Table 1 A 4*10 dimensional matrix M1 to be compressed can be obtained according to Table 1:

[0075]

[0076] The specific process of using the word vector matrix compression method based on clustering and principal component analysis is as follows:

[0077] First, the words in the compression matrix M1 are clustered, such as k-means clustering, hierarchical clustering and other algorithms, and the Euclidean distance between word vectors is used as a similarity measurement indicator for classification.

[0078] Continuing with the example, for Table 1, the hyperparameter in the clustering algorithm, that is, the number of categories k, can be set to 2. Since the Euclidean distance of the word vectors corresponding to "happy" and "joy" is much smaller than the Euclidean distance of the word vectors corresponding to "happy" and "sad", "happy" and "joy" can be classified into one category. Similarly, "sad" and "sadness" can be classified into another category based on the Euclidean distance. The clustering results can be shown in Table 2 below.

[0079]

[0080]

[0081] Table 2

[0082] Secondly, select the representative word and category center of each category from the clustering results. Usually, words with a preset frequency can be selected as representative words, that is, common words, and the category center is the word vector corresponding to the representative word.

[0083] Continuing with the example, the selected representative words and category centers can be shown in Table 3 below:

[0084] category Representative words Word vectors 1 Happy [0.8966,0.8534,…,0.9001] 2 sad [0.1731,0.2419,…,0.3453]

[0085] Table 3 At this point, the word vectors in Table 3 can be represented by matrix M2:

[0086]

[0087] Among them, matrix M2 is a new word vector matrix obtained by compressing the rows of the word vectors of matrix M1 through the clustering algorithm. In matrix M2, the length of each word vector is still 10, and the number of columns of the word vectors needs to be further compressed.

[0088] Next, principal component analysis is performed on the features contained in the initially compressed word vectors, and the final compressed matrix M3 is obtained by selecting the first n principal components.

[0089] Continuing with the example, suppose that after principal component analysis is performed on the word vector of M2, the matrix M3 obtained by selecting the first 5 principal components is:

[0090]

[0091] Then Table 3 can be updated as shown in Table 4 below:

[0092] category Representative words Word vectors 1 Happy [0.7845,0.8435,0.8642,0.6983,0.9300] 2 sad [0.2312,0.3892,0.4319,0.3875,0.2488]

[0093] Table 4

[0094] That is, matrix M3 is the final compressed matrix obtained based on clustering and principal component analysis. At this time, the word vector matrix to be stored is compressed from 4*10 dimensional M1 to 2*5 dimensional M3.

[0095] From the above process, we can imagine the importance of vocabulary compression when facing a huge vocabulary in a language. When compressing word vectors from the two perspectives of clustering and principal component analysis, on the one hand, due to the richness and diversity of the vocabulary in the language, it is difficult to select the hyperparameter k in the clustering algorithm, so that when compressing the rows of word vectors, the clustering results may not be accurate, and because the total vocabulary after clustering is reduced, the expressive power of the compressed vocabulary is limited;

[0096] On the other hand, when compressing word vector columns, compression based on principal component analysis is essentially a linear dimensionality reduction technology, which is usually suitable for situations where the spatial distribution is Gaussian. However, the distribution of vocabulary in actual language is usually highly nonlinear, resulting in the results based on principal component analysis not necessarily being accurate and not being the optimal solution for selecting word vector elements.

[0097] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0099] As mentioned above, the present inventors have discovered that when a clustering algorithm is used to perform row compression of word vectors, the compression is performed by reducing the vocabulary size, which results in limited expressiveness of the compressed word vectors. When a principal component analysis is used to perform column compression of word vectors, the linear dimensionality reduction technique is used, which is not suitable for the highly nonlinear distribution of vocabulary in a language, resulting in inaccurate principal components.

[0100] In view of this, the present specification provides a technical solution for transforming word vectors into frequency domain vectors, and sorting them based on the size of the modulus value of the transformed frequency domain vectors, and then obtaining the key elements in the word vectors based on the sorting results, and then constructing new word vectors based on the key elements. In this way, without reducing the vocabulary, word vector compression is achieved for words with a highly nonlinear distribution, thereby reducing the maintenance cost of the word vectors.

[0101] The core technical concept of this manual is:

[0102] The word vector can be regarded as a signal wave of finite length, and each element can be regarded as the signal sampling value in discrete time obtained after equally spaced sampling of the signal in the continuous time domain. After using Fourier transform, the discrete signal in the time domain can be converted into a discrete signal in the frequency domain.

[0103] The word vector regarded as a signal wave is transformed into the frequency domain space through Fourier transform to obtain a frequency domain vector of the same length. The modulus value of each element of the frequency domain vector is then calculated and sorted based on the size of the modulus value, so as to select several elements to construct the compressed word vector.

[0104] Through the above technical scheme, on the one hand, since the vocabulary size is not reduced when compressing the word vector, the original expression ability of the word vector is not changed; on the other hand, since the word vector is transformed into the frequency domain space, it can be sorted based on the size of the modulus value of each element in the frequency domain vector, so as to realize sorting according to the importance of each element of the word vector. Therefore, it is not only able to process the vocabulary in the language with highly nonlinear distribution, but also can remove unimportant elements and retain only the elements representing the key information in the word vector, so as to realize the compression of the word vector and reduce the maintenance cost of the word vector.

[0105] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0106] Application Scenario Overview

[0107] The core of natural language processing tasks is to represent words through language models, such as the meaning of words, the semantics of words in context, etc. Word vectors can be used to represent the feature information of words. Usually, word vectors can be expressed in the form of vectors as shown in Table 1. Multiple word vectors can form a word vector matrix in the form of M1.

[0108] Therefore, the word vector matrix constructed based on the massive corpus can be widely used in natural language processing scenarios. For example, based on the songs collected by the user, other songs with similar lyrics and semantics to the song can be recommended to the user; for example, based on the user's comments on a certain piece of information in the information client, the user's emotions can be determined and collected, and so on.

[0109] However, due to the huge vocabulary and complex grammar in the language, a large number of features are often required to describe it, resulting in the generated word vector matrix being huge and occupying a considerable storage space. Therefore, it is necessary to provide an effective word vector matrix compression method.

[0110] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0111] Exemplary Methods

[0112] The technical concept of this specification will be described in detail below through specific embodiments.

[0113] The present disclosure aims to provide a technical solution for transforming word vectors into frequency domain vectors, sorting them based on the size of the modulus of the transformed frequency domain vectors, obtaining key elements in the word vectors based on the sorting results, and then constructing new word vectors based on the key elements, thereby achieving word vector compression for words with highly nonlinear distribution without reducing the vocabulary size, thereby reducing the maintenance cost of the word vectors.

[0114] In implementation, the word vector to be compressed may be subjected to Fourier transform to obtain a frequency domain vector corresponding to the word vector;

[0115] For example, the word vector to be compressed can be regarded as a signal wave, and the word vector to be compressed can be transformed into the frequency domain space through discrete Fourier transform or fast Fourier transform to obtain a frequency domain vector corresponding to the word vector.

[0116] After obtaining the frequency domain vector, the modulus of each element in the frequency domain vector may be calculated, and the elements in the frequency domain vector may be sorted based on the numerical values ​​of the modulus values;

[0117] For example, the modulus value may be calculated for each element in the frequency domain vector, and the elements in the frequency domain vector may be sorted from large to small according to the numerical values ​​of the calculated modulus values.

[0118] After the sorting is completed, a number of elements may be selected from the sorted frequency domain vector, and a compressed word vector may be constructed based on the selected elements;

[0119] For example, when the above-mentioned sorting strategy is to sort the values ​​of the modulus from large to small, several elements at the front of the sorting result can be selected; at the same time, the corresponding position information of these several elements in the frequency domain vector before sorting can also be obtained; further, the position information corresponding to several elements and the modulus values ​​of several elements can be used as new elements respectively, and the compressed word vector can be constructed based on the sorting of the numerical values ​​of the modulus values ​​of several elements.

[0120] Through the above technical scheme, on the one hand, since the vocabulary size is not reduced when compressing the word vector, the original expression ability of the word vector is not changed; on the other hand, since the word vector is transformed into the frequency domain space, it can be sorted based on the size of the modulus value of each element in the frequency domain vector, so as to realize sorting according to the importance of each element of the word vector. Therefore, it is not only able to process the vocabulary in the language with highly nonlinear distribution, but also can remove unimportant elements and retain only the elements representing the key information in the word vector, so as to realize the compression of the word vector and reduce the maintenance cost of the word vector.

[0121] See also Figure 1 , Figure 1 This is a flowchart of a word vector compression method based on frequency domain transformation provided by an exemplary embodiment, the method comprising the following steps:

[0122] Step 101, performing Fourier transform on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector.

[0123] The above word vectors to be compressed can be represented by floating point numbers. For example, "apple" can be represented as [0.343434, -0.88749, 0.992112, ..., 0.232432].

[0124] Assuming that the vector length of the above-mentioned word vector to be compressed is N, the word vector can be regarded as a signal wave, and each element can be regarded as a signal sampling value in discrete time obtained after equally spaced sampling of the signal in the continuous time domain.

[0125] For example, the word vector [X0, X1, ..., X N-1 ] T Each element in represents the signal value at time 0, 1, ..., N-1.

[0126] Furthermore, through Fourier transform, the word vector regarded as a signal wave is transformed into the frequency domain space to obtain a frequency domain vector with the same length.

[0127] Continuing with the example, the frequency domain vector corresponding to the above word vector can be [Y0, Y1, ..., Y N-1 ] T .

[0128] It is worth noting that each element in the frequency domain vector is in complex form, that is, Y k =a k +b k i,i 2 =-1, 0≤k≤N-1.

[0129] Specifically,

[0130] Among them, i represents the imaginary unit, that is,

[0131] In one embodiment shown, the Fourier transform algorithm used to perform Fourier transform on the word vector to be compressed may include discrete Fourier transform;

[0132] Furthermore, the word vector to be compressed may be subjected to a discrete Fourier transform based on the following formula:

[0133]

[0134] Among them, [X0, X1, …, X N-1 ] T represents the word vector to be compressed, [Y0, Y1, ..., Y N-1 ] T represents the frequency domain vector, Y k =a k +b k i,i 2 =-1, 0≤k≤N-1, W N represents a factor matrix of size N×N,

[0135]

[0136] Among them, ω N Represents the factor, ω N =e^(-2πi / N).

[0137] In computer processing, reducing time complexity can speed up the algorithm and improve its efficiency. From the calculation formula of discrete Fourier transform, we can know that the time complexity of discrete Fourier transform is O(N 2 ), in order to further reduce the time complexity, the fast Fourier transform can be used to reduce the amount of calculation by continuously dividing the word vector into two new word vectors, and the length of the new word vector is half of the original length. The time complexity of the fast Fourier transform is O(NlogN).

[0138] For example, assuming N = 8, the word vector to be compressed, that is, the time domain vector [X0, X1, ..., X7] T Convert to frequency domain vector [Y0, Y1, ..., Y7] T The calculation formula for the discrete Fourier transform of is:

[0139]

[0140] Among them, ω N =e^(-2πi / N).

[0141] Since the above function contains exponential function and unit imaginary number, we can refer to Euler formula e ix =cosx+isinx, we get the following properties:

[0142]

[0143]

[0144] Periodicity:

[0145] symmetry:

[0146] Based on the above properties and divided according to the odd and even terms, the first half and the second half of the frequency domain vector can be calculated separately:

[0147]

[0148] in,

[0149]

[0150]

[0151] From the above formula, we can see that the discrete Fourier transform of a time domain vector of length 8 can be converted into two vectors of length 4 for discrete Fourier transform respectively; if the odd terms and even terms are further divided, it is equivalent to performing discrete Fourier transform on four vectors of length 2 respectively; further division can reduce the length of the vector to be discrete Fourier transformed to 1.

[0152] It can be seen that by dividing the word vector into odd and even items in turn, the length of the word vector is gradually reduced until it is 1. In this process, it only needs to be performed logN times. After each time the Fourier transform corresponding to two vectors of length N / 2 is obtained, we can obtain the Fourier transform corresponding to the vector of length N in O(N) time, so the total time complexity is only O(NlogN).

[0153] For example, if the word vector length N = 200, the time complexity of using discrete Fourier transform is 40,000, while the time complexity of using fast Fourier transform is reduced to 460, a reduction of almost two orders of magnitude.

[0154] In another embodiment shown, the Fourier transform algorithm used to perform Fourier transform on the word vector to be compressed may include fast Fourier transform;

[0155] Furthermore, the word vector to be compressed is subjected to a fast Fourier transform based on the following formula:

[0156]

[0157] in, Indicates size The identity matrix of Indicates size The diagonal matrix of

[0158]

[0159] Indicates size The factor matrix of

[0160]

[0161] in, Represents the factor,

[0162] It is worth noting that the above matrix transformation is an equivalent form of the aforementioned fast Fourier transform.

[0163] In addition, since the fast Fourier transform needs to continuously divide the odd and even terms, N needs to satisfy 2 m , m needs to be a positive integer.

[0164] Therefore, in one example, before performing the fast Fourier transform, it is necessary to first determine whether the length N of the word vector to be compressed satisfies N=2. m ;

[0165] If satisfied, perform fast Fourier transform;

[0166] If not, you need to make sure that N < 2 is satisfied. m The minimum value of m is (2 m -N) 0s are filled into the word vector to be compressed as new elements, and a fast Fourier transform is performed on the filled word vector.

[0167] Both the discrete Fourier transform and the fast Fourier transform can realize the conversion of word vectors to frequency domain vectors, and technicians in this field can choose according to actual needs.

[0168] Step 102: Calculate the modulus of each element in the frequency domain vector, and sort the elements in the frequency domain vector based on the numerical values ​​of the modulus.

[0169] From the above explanation, we can know that any signal wave can be decomposed into the superposition and or integration of a series of sine waves. Using Fourier transform, we can find the sine wave components of each frequency contained in a discrete-time signal of a finite length. Each element in the frequency domain vector obtained after Fourier transforming the time domain vector can be expressed in the form of complex numbers. These complex numbers represent the amplitudes corresponding to each sinusoidal wave component in the frequency domain space. Obviously, the larger the amplitude, the greater the influence of the sine wave corresponding to the amplitude.

[0170] Therefore, the amplitude of each sine wave can be determined by calculating the modulus value of each element in the frequency domain vector, and the elements in the frequency domain vector can be sorted based on the numerical value of the modulus value.

[0171] It is worth noting that numbers of the form z = a + bi are called complex numbers, where i is the imaginary unit, i 2 = -1, a, b are arbitrary real numbers, a is the real part of the complex number, b is the imaginary part of the complex number, and the modulus of the complex number z is

[0172] Step 103, selecting a number of elements from the sorted frequency domain vector, and constructing a compressed word vector based on the selected elements.

[0173] In this specification, after the elements in the frequency domain vector are sorted based on the numerical values ​​of the modulus values, several elements can be selected from the sorted frequency domain vector according to a preset selection rule as elements for constructing a new word vector.

[0174] For example, several elements with the largest modulus values ​​can be selected as new elements to construct a compressed word vector. The specific number of elements can be determined by technicians in this field based on actual needs, and this disclosure does not limit this.

[0175] In one embodiment shown, when the sorting strategy is to sort the values ​​of the module values ​​from large to small, several front elements in the sorting result are selected.

[0176] For example, in one example, the calculation results of the modulus values ​​may be arranged from large to small in numerical value, and the first Q elements may be selected from the one-dimensional array;

[0177] Of course, in another example, the calculation results of the modulus values ​​may be arranged in ascending order of numerical value, and the last Q elements may be selected from the one-dimensional array.

[0178] In an illustrated implementation manner, position information corresponding to the plurality of elements in the frequency domain vector before sorting may also be obtained.

[0179] For example, assuming that the values ​​of the modulus values ​​are sorted from large to small, the top five elements are selected, and the corresponding position information of these five elements in the frequency domain vector before sorting is obtained.

[0180] In an example, since the elements in the frequency domain vector can be represented by Yk=ak+bki, that is, k can represent the corresponding position information of each element in the frequency domain vector before sorting.

[0181] Continuing with the example, assuming that the five elements selected above are sorted from large to small according to the modulus value, they are Y12, Y33, Y1, Y4 and Y91 respectively. Then the corresponding position information of each element in the frequency domain vector before sorting is 12, 33, 1, 4 and 91 respectively.

[0182] In one embodiment shown, the compressed word vector can be constructed based on any of the following methods:

[0183] In one example, the position information corresponding to the plurality of elements and the modulus values ​​of the plurality of elements may be used as new elements respectively, and the compressed word vector may be constructed based on the sorting of the plurality of elements;

[0184] For example, for the elements Y12, Y33, Y1, Y4 and Y91 in the determined frequency domain vector, the position information 12, 33, 1, 4 and 91 corresponding to the five elements and the modulus values ​​of the five elements can be used as the basis. and As new elements respectively, and based on the sorting results of these five elements by modulus value, construct the compressed word vector shown below:

[0185]

[0186] In another example, the position information corresponding to the plurality of elements and the square of the modulus values ​​of the plurality of elements may be used as new elements respectively, and the compressed word vector may be constructed based on the sorting of the plurality of elements;

[0187] For example, for the elements Y12, Y33, Y1, Y4 and Y91 in the determined frequency domain vector, the position information 12, 33, 1, 4 and 91 corresponding to the five elements and the square of the modulus values ​​of the five elements can be used to calculate the value of and As new elements respectively, and based on the sorting results of these five elements by modulus value, construct the compressed word vector shown below:

[0188]

[0189] In another example, the position information corresponding to the elements, the real parts of the elements, and the imaginary parts of the elements may be respectively used as new elements, and the compressed word vector may be constructed based on the order of the elements.

[0190] For example, for the elements Y12, Y33, Y1, Y4, and Y91 in the determined frequency domain vector, the position information 12, 33, 1, 4, and 91 corresponding to these five elements, as well as the real and imaginary parts of these five elements can be used as new elements respectively, and based on the sorting results of these five elements according to the size of the modulus value, the compressed word vector shown below can be constructed:

[0191]

[0192] Since the length of the word vector to be compressed is generally very large, the value of the element representing the position information in the above reconstructed word vector may be too large. The position information can be processed and then used to construct the compressed word vector.

[0193] In one illustrated embodiment, the position information includes the reciprocal of position information corresponding to the plurality of elements in the frequency domain vector before sorting.

[0194] For example, for the elements Y12, Y33, Y1, Y4 and Y91 in the determined frequency domain vector, the position information can be taken as 1 / 12, 1 / 33, 1, 1 / 4 and 1 / 91, and used to construct the compressed word vector. The construction method can refer to the above three methods, which will not be repeated here.

[0195] In the above embodiment, the word vector regarded as a signal wave is transformed into the frequency domain space through Fourier transform to obtain a frequency domain vector of the same length, and then the modulus of each element of the frequency domain vector is calculated, and sorted based on the size of the modulus, so as to select several elements to construct a compressed word vector. Through the above technical scheme, on the one hand, since the vocabulary is not reduced when compressing the word vector, the original expression ability of the word vector is not changed; on the other hand, since the word vector is transformed into the frequency domain space, it can be sorted based on the size of the modulus of each element in the frequency domain vector, and the importance of each element of the word vector can be sorted. Therefore, it can not only process the vocabulary in the language with highly nonlinear distribution, but also remove unimportant elements, and only retain the elements representing the key information in the word vector, so as to achieve compression of the word vector and reduce the maintenance cost of the word vector.

[0196] Exemplary Devices

[0197] After introducing the method of the exemplary embodiment of the present disclosure, next, see Figure 2 , Figure 2It is a block diagram of a word vector compression device based on frequency domain transformation provided by an exemplary embodiment.

[0198] The implementation process of the functions and effects of each module in the following device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here. For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment.

[0199] like Figure 2 As shown, the word vector compression device 200 based on frequency domain transformation may include: a transformation module 201, a sorting module 202 and a construction module 203. Among them:

[0200] The transformation module 201 is configured to perform Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector;

[0201] The sorting module 202 is configured to calculate the modulus value of each element in the frequency domain vector, and sort the elements in the frequency domain vector based on the numerical value of the modulus value;

[0202] The construction module 203 is configured to select a number of elements from the sorted frequency domain vector, and construct a compressed word vector based on the selected elements.

[0203] In one embodiment, the Fourier transform algorithm used to perform Fourier transform on the word vector to be compressed includes discrete Fourier transform;

[0204] The transformation module 201 further:

[0205] The word vector to be compressed is subjected to discrete Fourier transform based on the following formula:

[0206]

[0207] Among them, [X0, X1, …, X N-1 ] T represents the word vector to be compressed, [Y0, Y1, ..., Y N-1 ] T represents the frequency domain vector, Y k =a k +b k i,i 2 =-1, 0≤k≤N-1, W N represents a factor matrix of size N×N,

[0208]

[0209] Among them, ω N Represents the factor, π N=e^(-2πi / N).

[0210] In one embodiment, the Fourier transform algorithm used to perform Fourier transform on the word vector to be compressed includes fast Fourier transform;

[0211] The transformation module 201 further:

[0212] The word vector to be compressed is subjected to a fast Fourier transform based on the following formula:

[0213]

[0214] in, Indicates size The identity matrix of Indicates size The diagonal matrix of

[0215]

[0216] Indicates size The factor matrix of

[0217]

[0218] in, Represents the factor,

[0219] In one embodiment, the construction module 203 further:

[0220] When the sorting strategy is to sort the values ​​of the module values ​​from large to small, select the first several elements in the sorting result.

[0221] In one embodiment, the apparatus 200 further includes:

[0222] The position module 204 obtains the position information corresponding to the plurality of elements in the frequency domain vector before sorting.

[0223] In one embodiment, the construction module 203 further:

[0224] Taking the position information corresponding to the plurality of elements and the modulus values ​​of the plurality of elements as new elements respectively, and constructing the compressed word vector based on the sorting of the plurality of elements;

[0225] or,

[0226] Taking the position information corresponding to the plurality of elements and the square of the modulus values ​​of the plurality of elements as new elements respectively, and constructing the compressed word vector based on the sorting of the plurality of elements;

[0227] or,

[0228] The position information corresponding to the elements, the real parts of the elements, and the imaginary parts of the elements are respectively used as new elements, and the compressed word vector is constructed based on the sorting of the elements.

[0229] In one embodiment, the position information includes the inverse of position information corresponding to the plurality of elements in the frequency domain vector before sorting.

[0230] The specific details of each module of the above-mentioned word vector compression device 200 based on frequency domain transformation have been described in detail in the previous description of the word vector compression method process based on frequency domain transformation, so they will not be repeated here.

[0231] It should be noted that although several modules or units of the word vector compression device 200 based on frequency domain transformation are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0232] Exemplary Media

[0233] After introducing the apparatus of the exemplary embodiment of the present disclosure, next, see Figure 3 , Figure 3 It is a schematic diagram of a word vector compression medium based on frequency domain transformation provided by an exemplary embodiment.

[0234] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0235] refer to Figure 3 As shown, a program product 30 for implementing the above method according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0236] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0237] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0238] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0239] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0240] Exemplary Computing Devices

[0241] After introducing the method, medium and apparatus of the exemplary embodiments of the present disclosure, see Figure 4 , Figure 4It is a schematic diagram of an electronic device capable of implementing the above method provided by an exemplary embodiment.

[0242] Refer to the following Figure 4 hereinafter describes an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0243] like Figure 4 As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: at least one processing unit 401, at least one storage unit 402, and a bus 403 connecting different system components (including the storage unit 402 and the processing unit 401).

[0244] The storage unit stores program codes, and the program codes can be executed by the processing unit 401, so that the processing unit 401 executes the steps of the above-mentioned various embodiments of this specification.

[0245] The storage unit 402 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 4021 and / or a cache storage unit 4022 , and may further include a read-only storage unit (ROM) 4023 .

[0246] The storage unit 402 may also include a program / utility 4024 having a set (at least one) of program modules 4025, such program modules 4025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include the reality of a network environment.

[0247] Bus 403 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0248] The electronic device 400 may also communicate with one or more external devices 404 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 405. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 406. As shown, the network adapter 406 communicates with other modules of the electronic device 400 via a bus 403. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0249] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0250] It should be noted that, although several units / modules or subunits / modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0251] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0252] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A word vector compression method based on frequency domain transformation, comprising: Performing Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector; Calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical values ​​of the modulus; When the sorting strategy is to sort the values ​​of the modulus values ​​from large to small, select a number of elements that are at the front of the sorting result; Constructing a compressed word vector based on the selected elements, including: taking position information corresponding to the elements and the modulus values ​​of the elements as new elements, and constructing the compressed word vector based on the sorting of the elements; The position information corresponding to the plurality of elements in the frequency domain vector before sorting is obtained, wherein the position information includes the inverse of the position information corresponding to the plurality of elements in the frequency domain vector before sorting.

2. According to the method of claim 1, the Fourier transform algorithm used in performing Fourier transform on the word vector to be compressed comprises discrete Fourier transform; The Fourier transforming of the word vector to be compressed includes: The word vector to be compressed is subjected to discrete Fourier transform based on the following formula: Among them, [X0,X1,…,X N-1 ] T represents the word vector to be compressed, [Y0,Y1,…,Y N-1 ] represents the frequency domain vector, Y k =a k +b k i,i 2 =-1, 0≤k≤N-1, W N represents a factor matrix of size N×N, Among them, ω N Represents the factor, ω N =e^(-2πi / N).

3. According to the method of claim 1, the Fourier transform algorithm used in performing Fourier transform on the word vector to be compressed comprises a fast Fourier transform; The Fourier transforming of the word vector to be compressed includes: The word vector to be compressed is subjected to a fast Fourier transform based on the following formula: in, Indicates size The identity matrix of Indicates size The diagonal matrix of Indicates size The factor matrix of in, Represents the factor, 4. The method according to claim 1, wherein the step of constructing a compressed word vector based on the selected elements further comprises: Taking the position information corresponding to the plurality of elements and the square of the modulus values ​​of the plurality of elements as new elements respectively, and constructing the compressed word vector based on the sorting of the plurality of elements; or, The position information corresponding to the elements, the real parts of the elements, and the imaginary parts of the elements are respectively used as new elements, and the compressed word vector is constructed based on the sorting of the elements.

5. A word vector compression device based on frequency domain transformation, comprising: A transformation module performs Fourier transformation on the word vector to be compressed to obtain a frequency domain vector corresponding to the word vector; A sorting module, calculating the modulus of each element in the frequency domain vector, and sorting the elements in the frequency domain vector based on the numerical values ​​of the modulus; A construction module, when the sorting strategy is to sort the values ​​of the modulus values ​​from large to small, selects a number of front elements in the sorting result, and constructs a compressed word vector based on the selected elements, specifically for taking the position information corresponding to the elements and the modulus values ​​of the elements as new elements, respectively, and constructing the compressed word vector based on the sorting of the elements; A position module is used to obtain position information corresponding to the plurality of elements in the frequency domain vector before sorting, wherein the position information includes the inverse of the position information corresponding to the plurality of elements in the frequency domain vector before sorting.

6. A medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 4.

7. A computing device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 4 by running the executable instructions.