Text representation method, word representation method, corresponding apparatus, medium and device

Abstract

The present disclosure relates to a text representation method, a word representation method, a corresponding device, a medium and equipment, the text representation method comprising: obtaining a text and performing word segmentation on the text to obtain a word sequence; generating a text vector of the text, and performing at least one round of iteration on the text vector, each round of iteration comprising multiple sub-iterations; in the i-th sub-iteration of each round of iteration, based on a sliding window with a length of k, after sliding the sliding window of the last sub-iteration on the word sequence backward, performing weighted combination on the real word vector of the k words in the current sliding window and the text vector optimized in the last sub-iteration, taking the vector after the weighted combination as the predicted word vector of the next word outside the current sliding window; optimizing the text vector according to the predicted word vector and the real word vector of the next word; and after completing the at least one round of iteration, obtaining a final text vector for representing the text. The present disclosure can accurately obtain the vector representation of the text.

Description

Technical Field

[0001] This disclosure relates to the field of text processing technology, specifically to a text representation method, a word representation method, a corresponding device, medium, and equipment. Background Technology

[0002] With the explosive growth of information, manually annotated data has become time-consuming, of low quality, and easily influenced by the subjective opinions of the annotators. Therefore, automating text classification using machines has become practically significant. Delegating repetitive and tedious text annotation tasks to computers can effectively overcome these problems, while the classified data is consistent and of high quality. Text classification tasks are commonly used in digital libraries, public opinion analysis, news recommendation, email filtering, and other fields, providing strong support for the querying and retrieval of text resources.

[0003] Text classification relies on vector representations of text. Current mainstream algorithms use convolutional neural networks (CNNs), which first represent the text as a matrix, then extract features using the CNN, and finally input these feature vectors into a classifier for text classification. These algorithms require truncating each piece of text to the same length to fit the input of the CNN, which leads to a loss of accuracy. Furthermore, each operation of the CNN can only consider local information of the text and cannot accurately represent the entire text. Summary of the Invention

[0004] The purpose of this disclosure is to provide a text representation method, a word representation method, a corresponding device, medium, and equipment for accurately representing text.

[0005] To achieve the above objectives, firstly, this disclosure provides a text representation method, including:

[0006] Obtain the first target text and perform word segmentation on the first target text to obtain a word sequence;

[0007] Generate a text vector of the first target text, and perform at least one round of iteration on the text vector, each round of iteration containing multiple sub-iterations;

[0008] In the i-th sub-iteration of each iteration, based on a sliding window of length k, after sliding the sliding window of the previous sub-iteration on the word sequence backward, the real word vectors of the k words in the current sliding window and the optimized text vectors of the previous sub-iteration are weighted and combined, and the weighted and combined vector is used as the predicted word vector of the next word outside the current sliding window.

[0009] The text vector is optimized based on the predicted word vector and the actual word vector of the next word;

[0010] After completing at least one round of iterations, a final text vector representing the first target text is obtained.

[0011] Optionally, the actual word vector for each word is obtained as follows:

[0012] The second target text is obtained and segmented into multiple words;

[0013] Each word is treated as a node, and the nodes corresponding to adjacent words in the second target text are connected to generate an undirected graph;

[0014] Each node is taken as a starting node, and a random walk is performed on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, resulting in multiple sets of node sequences.

[0015] Combine the nodes in each node sequence in order to obtain multiple node pairs;

[0016] Generate word vectors for each node, and use the similarity of word vectors for two nodes in the same node pair in vector representation as the optimization objective to iteratively optimize the word vectors for each node.

[0017] After the iterative optimization is completed, the final word vector representing each word is obtained, and the final word vector of each word is used as the true word vector.

[0018] Optionally, the step of iteratively optimizing the word vectors corresponding to each node by using the similarity in vector representation between the word vectors corresponding to the two nodes in the same node pair as the optimization objective includes:

[0019] In each iteration, the similarity between two word vectors in the vector representation is calculated based on the word vectors corresponding to the two nodes in each node pair.

[0020] Calculate the loss value based on the similarity corresponding to each node;

[0021] Based on the loss value, the word vector corresponding to each node is updated using gradient descent.

[0022] Optionally, the similarity between two word vectors in their vector representations is calculated using the following formula:

[0023]

[0024] Among them, v i and v j This represents two nodes in the same node pair. Represents node v i Word vectors, Represents node vj The word vector, p(v i ,v j ) represents a node pair (v i ,v j The similarity in vector representation between the word vectors corresponding to the two nodes in the equation.

[0025] Optionally, the loss value is calculated using the following objective function:

[0026]

[0027] Where J represents the objective function, w ij Represents a pair of nodes (v) i ,v j The cosine similarity between the word vectors corresponding to the two nodes in the equation is denoted by log, where log represents the logarithmic function and λ is the regularization factor.

[0028] Optionally, generating the text vector of the first target text includes:

[0029] Based on the vector dimensions of the real word vectors, generate the text vectors of the same dimensions.

[0030] Secondly, this disclosure provides a word representation method, including:

[0031] The target text is obtained and segmented into multiple words.

[0032] Each word is treated as a node, and the nodes corresponding to adjacent words in the target text are connected to generate an undirected graph;

[0033] Each node is taken as a starting node, and a random walk is performed on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, resulting in multiple sets of node sequences.

[0034] Combine the nodes in each node sequence in order to obtain multiple node pairs;

[0035] Generate word vectors for each node, and use the similarity of word vectors for two nodes in the same node pair in vector representation as the optimization objective to iteratively optimize the word vectors for each node.

[0036] After the iterative optimization is completed, the final word vector representing each word is obtained.

[0037] Thirdly, this disclosure provides a text representation device, comprising:

[0038] The first word segmentation module is used to acquire the first target text and segment the first target text into words to obtain a word sequence;

[0039] The text vector iteration module is used to generate the text vector of the first target text and perform at least one round of iteration on the text vector, each round of iteration containing multiple sub-iterations; in the i-th sub-iteration of each round of iteration, based on a sliding window of length k, after sliding the sliding window of the previous sub-iteration on the word sequence backward, the real word vectors of the k words in the current sliding window and the text vector optimized by the previous sub-iteration are weighted and combined, and the weighted and combined vector is used as the predicted word vector of the next word outside the current sliding window. Based on the predicted word vector and the real word vector of the next word, the text vector is optimized.

[0040] The text vector acquisition module is used to obtain a final text vector representing the first target text after completing the at least one round of iterations.

[0041] Fourthly, this disclosure provides a word representation device, comprising:

[0042] The second word segmentation module is used to acquire the target text and segment the target text into multiple words;

[0043] The graph generation module is used to treat each word as a node and connect the nodes corresponding to adjacent words in the target text to generate an undirected graph.

[0044] The random walk module is used to take each node as a starting node and perform a random walk on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, and a total of multiple sets of node sequences are obtained.

[0045] The node combination module is used to combine nodes in each node sequence in pairs in order to obtain multiple node pairs.

[0046] The word vector iteration module is used to generate word vectors corresponding to each node, and uses the similarity of the word vectors corresponding to the two nodes in the same node pair in the vector representation as the optimization objective to iteratively optimize the word vectors corresponding to each node.

[0047] The word vector acquisition module is used to obtain the final word vector representing each word after the iterative optimization is completed.

[0048] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first or second aspect.

[0049] Sixthly, this disclosure provides an electronic device, including:

[0050] A memory on which computer programs are stored;

[0051] A processor for executing the computer program in the memory to implement the steps of the method described in the first or second aspect.

[0052] In the above scheme, a text vector for the first target text is first initialized and generated. This text vector is then optimized through continuous iteration. The iteration process includes one or more rounds, each containing multiple sub-iterations. In each sub-iteration, only a small subset of words from the word sequence is slid out to participate in the text vector optimization. As the sliding window continues to slide, the text vector is optimized based on word order and global information. It should be understood that the text vector is shared across all sub-iterations of the same text. The input for each sub-iteration includes the text vector, which can be considered as the main idea of ​​the text. Through the text vector, the main idea of ​​the text is used as part of the input for iterative optimization each time. Thus, as k words are slid out from the word sequence for each iteration, the main idea expressed by the text vector becomes increasingly accurate as a shared parameter for each iteration, ultimately leading to an accurate vector representation of the text. Furthermore, the above scheme naturally solves the problem of inconsistent text lengths.

[0053] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 A flowchart of a word representation method in an exemplary embodiment is shown;

[0056] Figure 2 A schematic diagram of an undirected graph generated in an exemplary embodiment is shown;

[0057] Figure 3 A flowchart illustrating a specific implementation of step S105 in an exemplary embodiment is shown;

[0058] Figure 4 A flowchart of a text representation method in an exemplary embodiment is shown;

[0059] Figure 5 A block diagram of a text representation device in an exemplary embodiment is shown;

[0060] Figure 6 A block diagram of a word representation apparatus in an exemplary embodiment is shown;

[0061] Figure 7A block diagram of an electronic device in an exemplary embodiment is shown. Detailed Implementation

[0062] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0063] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0064] First, this disclosure introduces a word representation method for accurately representing words in text using vectors. Figure 1 A flowchart of a word representation method in an exemplary embodiment is shown, with reference to Figure 1 The method includes:

[0065] S101, Obtain the target text and perform word segmentation on the target text to obtain multiple words.

[0066] The program retrieves one or more target texts and performs word segmentation on them to obtain multiple words. Specifically, after segmenting the target text, stop words are removed from the segmentation results to obtain the desired multiple words.

[0067] S102, treat each word as a node, and connect the nodes corresponding to adjacent words in the target text to generate an undirected graph.

[0068] After obtaining multiple words, each word is treated as a node, and the adjacency relationship between two words in the target text is used as the edge of the corresponding node to generate an undirected graph. The connection between nodes in the undirected graph indicates that the corresponding two words appear adjacently in the target text. For two words that do not appear adjacently in the target text, their corresponding nodes are not connected.

[0069] Figure 2 A schematic diagram of an undirected graph generated in an exemplary embodiment is shown, with reference to... Figure 2 The undirected graph contains nine nodes, numbered V1 to V9. A connection between node V1 and node V2 indicates that the word corresponding to node V1 and the word corresponding to node V2 are adjacent in the target text. Adjacent occurrence can mean that the two words are adjacent after removing stop words.

[0070] S103, take each node as the starting node, and perform a random walk on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, and obtain multiple sets of node sequences in total.

[0071] Each node is used as a starting node. For each starting node, a random walk is performed on the undirected graph, beginning at the starting node, until the walk ends, resulting in a sequence of nodes corresponding to that starting node. Multiple node sequences are obtained in total. The random walk process involves starting from the starting node, randomly selecting a node from among the nodes connected to the starting node as the next node, and then randomly selecting a node from among the nodes connected to the next node, repeating this process until the walk ends. The walk length can be preset based on the total length of the target text. The random walk ends when the preset walk length is reached.

[0072] For example, targeting Figure 2 In the undirected graph shown, assuming node V1 is the starting node, since only node V2 is connected to node V1, node V2 must be chosen as the next node. Since node V2 is connected to nodes V1, V3, and V5, a node is randomly selected from V1, V3, and V5 (let's assume V5 is chosen) as the next node. Since node V5 is connected to nodes V2 and V6, a node is randomly selected from V2 and V6 (let's assume V6 is chosen) as the next node. Since node V6 is connected to nodes V5, V7, and V8, a node is randomly selected from V5, V7, and V8 (let's assume V8 is chosen) as the next node. Since node V8 is connected to nodes V6 and V9, a node is randomly selected from V6 and V9 (let's assume V6 is chosen) as the next node. At this point, the walk length has reached the preset walk length, and the random walk ends. The output is a sequence of nodes starting from node V1:

[0073] V1, V2, V5, V6, V8, V6.

[0074] For each starting node, a random walk is performed in the manner described above, resulting in multiple sets of node sequences.

[0075] S104: Combine the nodes in each node sequence in pairs in order to obtain multiple node pairs.

[0076] After obtaining the node sequence, the nodes in the node sequence are combined in pairs according to their adjacent order to obtain multiple node pairs. The two nodes in the same node pair are adjacent in the node sequence.

[0077] For example, for the above node sequence, the following node pairs can be obtained:

[0078] (V1,V2), (V2,V5), (V5,V6), (V6,V8), (V8,V6).

[0079] S105 generates word vectors corresponding to each node, and uses the similarity in vector representation between word vectors corresponding to two nodes in the same node pair as the optimization objective to iteratively optimize the word vectors corresponding to each node.

[0080] First, initialize and generate word vectors for each node, ensuring that the word vectors for each node have the same vector dimension. Then, using the similarity in vector representation between the word vectors of two nodes in the same node pair as the optimization objective, iteratively optimize the word vectors for each node until the iteration ends.

[0081] S106, after the iterative optimization is completed, the final word vector representing each word is obtained.

[0082] In the above process, the vector dimension is first determined, such as 64 dimensions. A 64-dimensional vector of initialized random numbers is then used as the word vector for this node. The word vector for this node is then optimized through continuous iteration. It should be understood that for two nodes in the same node pair—that is, two adjacent words in the text—these two words are considered semantically similar. Therefore, during iteration, the optimization goal is to make the vector representations of these two words as similar as possible. As the iteration progresses, similar words will exhibit even greater similarity in their vector representations.

[0083] Through the above process, we can obtain the vector representation of each word in the text. It can be seen that the more frequently a word appears in the text, the higher the degree of its corresponding node, and the higher the probability that the random walk will select it. Therefore, it will appear in more node pairs, and its vector will play a more important role in the iterative optimization process. Furthermore, as seen from the above process, this method not only focuses on the local information of the text but also obtains more global information through random walks, resulting in a more accurate word vector representation.

[0084] Figure 3 A flowchart illustrating a specific implementation of step S105 in an exemplary embodiment is shown, with reference to... Figure 3 Step S105 includes:

[0085] S201. In each iteration, the similarity between two word vectors in vector representation is calculated based on the word vectors corresponding to the two nodes in each node pair.

[0086] Optionally, the similarity between two word vectors in their vector representations is calculated using the following formula:

[0087]

[0088] Among them, v i and v jThis represents two nodes in the same node pair. Represents node v i Word vectors, Represents node v j The word vector, p(v i ,v j ) represents a node pair (v i ,v j The similarity in vector representation between the word vectors corresponding to the two nodes in the equation.

[0089] Understandably, the above formula is used to... The result is mapped to a numerical range of 0 to 1, therefore p(v i ,v j ) can be used to represent node pairs (v i ,v j The similarity in vector representation of the word vectors corresponding to two nodes in ), p(v i ,v j The closer the value of ) is to 1, the more similar the two word vectors are in their vector representations; the closer it is to 0, the less similar the two word vectors are in their vector representations.

[0090] S202, calculate the loss value based on the similarity of each node pair in the vector representation.

[0091] Optionally, the loss value is calculated using the following objective function:

[0092]

[0093] Where J represents the objective function, w ij Represents a pair of nodes (v) i ,v j The cosine similarity between the word vectors corresponding to the two nodes in the equation is denoted by log, where log represents the logarithmic function and λ is the regularization factor.

[0094] S203, based on this loss value, update the word vector corresponding to each node through gradient descent.

[0095] Based on the loss value, it is determined whether to update the word vectors. If the loss value is high, exceeding the set first loss threshold, the word vectors corresponding to each node are updated through gradient descent. If the loss value is low, less than the set first loss threshold, the update of the word vectors of each node can be stopped, that is, the iteration ends.

[0096] Specifically, for any node v in an undirected graph, the word vector of node v is calculated by the objective function. The partial derivative of the word vector of that node. Perform gradient descent updates:

[0097]

[0098] Among them, the left side of the equation This represents the updated word vector, and the right side of the equation... This represents the word vector before the update. This represents the partial derivative of the objective function with respect to the word vector of that node, where α is the learning rate.

[0099] Repeat steps S201 to S203 above to continuously update the word vector corresponding to each node until the iteration ends. After the iteration ends, the final word vector representing each word is obtained.

[0100] This disclosure also provides a text representation method. Figure 4 A flowchart of a text representation method in an exemplary embodiment is shown below. Please refer to... Figure 4 The method includes:

[0101] S301, Obtain the first target text and perform word segmentation on the first target text to obtain a word sequence.

[0102] The first target text to be represented is obtained, and word segmentation is performed on the first target text to obtain a word sequence. Specifically, after segmenting the first target text, stop words are removed from the segmentation results, and the word sequence is obtained based on the remaining words.

[0103] S302, generate the text vector of the first target text, and perform at least one round of iteration on the text vector, each round of iteration containing multiple sub-iterations; in the i-th sub-iteration of each round of iteration, based on a sliding window of length k, after sliding the sliding window of the previous sub-iteration on the word sequence backward, the real word vectors of the k words in the current sliding window and the text vector optimized by the previous sub-iteration are weighted and combined, and the weighted and combined vector is used as the predicted word vector of the next word outside the current sliding window. Based on the predicted word vector and the real word vector of the next word, the text vector is optimized.

[0104] Where i and k are both positive integers.

[0105] In a specific embodiment, a loss function is pre-defined, which is a function of the predicted word vector, the true word vector, and the text vector. Given the predicted and true word vectors of the next word outside the current sliding window, the partial derivative of this loss function with respect to the text vector is calculated. Then, gradient descent is applied to update the text vector, optimizing it in a direction that minimizes the error between the predicted and true word vectors. The specific method of gradient descent is not detailed here.

[0106] In the above steps, a text vector for the first target text is first initialized. Then, based on a sliding window of length k, the text vector is continuously slid backward across the word sequence. Each slide performs a sub-iteration based on the k words in the current sliding window. The text vector is continuously optimized through multiple sub-iterations until the iteration ends. The number of iteration rounds is preset; it can be one or more rounds. The multiple sub-iterations that the sliding window undergoes from the first word to the last word in the word sequence constitute one round of iteration. After the last sub-iteration of a round is completed, the first sub-iteration of the next round begins.

[0107] In the first sub-iteration of the first iteration, a sliding window of length k is placed at the beginning of the word sequence. The k words in the current sliding window are the first to the kth words in the word sequence. The real word vectors of the k words in the current sliding window are obtained. The real word vectors of these k words are weighted and combined with the initialized text vector. The weighted vector is used as the predicted word vector of the (k+1)th word in the word sequence. The text vector is optimized based on the predicted word vector and the real word vector of the (k+1)th word.

[0108] When i > 1, in the i-th sub-iteration of the first iteration, the sliding window of the previous sub-iteration on the word sequence is slid backward. Specifically, it can be slid backward by the length of one or more words. Then, the true word vectors of the k words in the current sliding window are obtained. The true word vectors of these k words are weighted and combined with the text vector optimized in the previous sub-iteration. The weighted vector is used as the predicted word vector of the next word outside the current sliding window. Based on the predicted word vector and the true word vector of the next word, the text vector is optimized.

[0109] Repeat the above steps, and the next iteration can be started after one iteration is completed.

[0110] In the first sub-iteration of the next iteration, a sliding window of length k is placed at the beginning of the word sequence. The k words in the current sliding window are the first to the kth words in the word sequence. The true word vectors of the k words in the current sliding window are obtained. The true word vectors of these k words are combined with the optimized text vectors from the previous sub-iteration (the last sub-iteration of the previous iteration) in a weighted manner. The weighted vector is used as the predicted word vector of the (k+1)th word in the word sequence. Based on the predicted word vector and the true word vector of the (k+1)th word, the text vector is optimized.

[0111] The subsequent sub-iteration process can refer to the process of the i-th sub-iteration of the first round of iteration, and will not be repeated here.

[0112] To facilitate understanding, let's take k=3 as an example to illustrate the above process. First, initialize the text vector text0 to generate the first target text. In the first sub-iteration of the first iteration, obtain the real word vectors of the 1st to 3rd words in the word sequence, denoted as w1, w2, and w3 respectively. Then, perform a weighted combination of the text vector text0 with w1, w2, and w3 to obtain the weighted vector, i.e.:

[0113] λ1*text0+λ2*w1+λ3*w2+λ4*w3; where λ1, λ2, λ3, and λ4 are the weighted weights corresponding to the text vector and each real word vector, respectively.

[0114] The weighted vector is used as the predicted word vector of the 4th word in the word sequence. Then, based on the predicted word vector and the true word vector of the 4th word, the text vector text0 is optimized and the optimized text vector is denoted as text1.

[0115] In the next sub-iteration, the true word vectors of the 2nd to 4th words in the word sequence are obtained, denoted as w2, w3, and w4 respectively. The text vector text1 is then weighted and combined with w2, w3, and w4 to obtain the weighted vector, i.e.:

[0116] λ1*text1+λ2*w2+λ3*w3+λ4*w4;

[0117] The weighted vector is used as the predicted word vector of the 5th word in the word sequence. Then, based on the predicted word vector and the true word vector of the 5th word, the text vector text1 is optimized and the optimized text vector is denoted as text2.

[0118] In the next sub-iteration, the true word vectors of the 3rd to 5th words in the word sequence are obtained and denoted as w3, w4, and w5 respectively. The text vector text2 is weighted and combined with w3, w4, and w5. The weighted vector is used as the predicted word vector of the 6th word in the word sequence. Then, based on the predicted word vector and the true word vector of the 6th word, the text vector text2 is optimized and the optimized text vector is denoted as text3.

[0119] The subsequent sub-iteration process will not be repeated.

[0120] Specifically, when initializing and generating text vectors, text vectors of the same dimension as the real word vectors are generated so that the text vectors can be weighted and combined with the real word vectors.

[0121] S303, after completing at least one round of iteration, obtains the final text vector used to represent the first target text.

[0122] In the above process, the vector dimension of the true word vector is first determined, such as 64 dimensions. A 64-dimensional vector of initialized random numbers is then used as the text vector of the first target text. This text vector is then optimized through continuous iteration. The iteration process includes one or more rounds, each containing multiple sub-iterations. In each sub-iteration, only a small portion of the word sequence is slid out to participate in the text vector optimization. As the sliding window continues to slide, the text vector is optimized based on word order information and global information. It should be understood that the text vector is shared across all sub-iterations of the same text. The input of each sub-iteration includes the text vector, which can be seen as the main idea of ​​the text. Through the text vector, the main idea of ​​the text is used as part of the input for iterative optimization each time. Thus, as k words are slid out and iterated through the word sequence each time, the main idea expressed by the text vector becomes increasingly accurate as a shared parameter for each iteration, ultimately leading to an accurate vector representation of the text.

[0123] In addition, the above solution can also naturally solve the problem of inconsistent text lengths.

[0124] Understandably, after obtaining the final text vector representing the first target text, this vector can be input into the trained classification model. The classification model outputs a classification result for the first target text based on the input final text vector, such as classifying the first target text as entertainment news. Optionally, each training sample of the classification model can be the final text vector obtained based on this technical solution.

[0125] It should be noted that the text representation method provided in this disclosure can be applied to most text-related tasks, including but not limited to information retrieval, text classification, sentiment analysis, etc., and can be used in practical scenarios such as search engines, news classification, and reading recommendation.

[0126] Furthermore, in the above process, the actual word vector of each word can be obtained based on the word representation method provided in the aforementioned embodiments.

[0127] In a specific embodiment, the actual word vector of each word is obtained in the following way:

[0128] First, the second target text is obtained and segmented into multiple words. The second target text and the first target text can be the same text or different texts.

[0129] Then, each word is treated as a node, and the nodes corresponding to adjacent words in the second target text are connected to generate an undirected graph.

[0130] Then, each node is taken as the starting node, and a random walk is performed on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, resulting in multiple sets of node sequences.

[0131] Then, the nodes in each node sequence are combined in pairs in order to obtain multiple node pairs.

[0132] Then, word vectors corresponding to each node are generated, and the similarity of word vectors corresponding to two nodes in the same node pair in vector representation is used as the optimization objective to iteratively optimize the word vectors corresponding to each node.

[0133] Finally, after the iterative optimization is completed, the final word vector representing each word is obtained, and the final word vector of each word is used as the real word vector.

[0134] In this process, the word vectors corresponding to each node are iteratively optimized by taking the similarity of the word vectors corresponding to the two nodes in the same node pair in the vector representation as the optimization objective. In each iteration, the similarity of the two word vectors in the vector representation is calculated based on the word vectors corresponding to the two nodes in each node pair. Then, the loss value is calculated based on the similarity of the word vectors corresponding to each node pair in the vector representation. Finally, the word vectors corresponding to each node are updated by gradient descent based on the loss value.

[0135] Optionally, the similarity between two word vectors in their vector representations is calculated using the following formula:

[0136]

[0137] Among them, v i and v j This represents two nodes in the same node pair. Represents node v i Word vectors, Represents node v j The word vector, p(v i ,v j ) represents a node pair (v i ,v j The similarity between the word vectors corresponding to the two nodes in the equation.

[0138] Optionally, the loss value is calculated using the following objective function:

[0139]

[0140] Where J represents the objective function, w ij Represents a pair of nodes (v) i ,v jThe cosine similarity between the word vectors corresponding to the two nodes in the equation is denoted by log, where log represents the logarithmic function and λ is the regularization factor.

[0141] Understandably, the specific steps and technical effects of obtaining the true word vector for each word can be found in the explanation of the word representation method provided above, and will not be repeated here.

[0142] Figure 5 A block diagram of a text display device in an exemplary embodiment is shown. Please refer to... Figure 5 The text display device 400 includes:

[0143] The first word segmentation module 401 is used to acquire the first target text and segment the first target text to obtain a word sequence;

[0144] The text vector iteration module 402 is used to generate the text vector of the first target text and perform at least one round of iteration on the text vector, each round of iteration containing multiple sub-iterations; in the i-th sub-iteration of each round of iteration, based on a sliding window of length k, after sliding the sliding window of the previous sub-iteration on the word sequence backward, the real word vectors of the k words in the current sliding window and the text vector optimized by the previous sub-iteration are weighted and combined, and the weighted and combined vector is used as the predicted word vector of the next word outside the current sliding window. The text vector is optimized based on the predicted word vector and the real word vector of the next word.

[0145] The text vector acquisition module 403 is used to obtain a final text vector representing the first target text after completing the at least one round of iterations.

[0146] Optionally, the actual word vector for each word is obtained through the following module:

[0147] The word segmentation submodule is used to obtain the second target text and segment it into multiple words.

[0148] The graph generation submodule is used to treat each word as a node and connect the nodes corresponding to adjacent words in the second target text to generate an undirected graph.

[0149] The random walk submodule is used to take each node as a starting node and perform a random walk on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, and a total of multiple sets of node sequences are obtained.

[0150] The node combination submodule is used to combine the nodes in each node sequence in pairs in order to obtain multiple node pairs;

[0151] The word vector iteration submodule is used to generate the word vector corresponding to each node, and iteratively optimizes the word vector corresponding to each node by taking the similarity of the word vectors corresponding to the two nodes in the same node pair in the vector representation as the optimization objective.

[0152] The word vector acquisition submodule is used to obtain the final word vector representing each word after the iterative optimization is completed, and the final word vector of each word is used as the real word vector.

[0153] Optionally, the word vector iteration submodule includes:

[0154] The similarity calculation submodule is used to calculate the similarity between two word vectors in vector representation based on the word vectors corresponding to the two nodes in each node pair in each iteration;

[0155] The loss calculation submodule is used to calculate the loss value based on the similarity corresponding to each node.

[0156] The word vector update submodule is used to update the word vector corresponding to each node by gradient descent based on the loss value.

[0157] Optionally, the similarity calculation submodule is used to calculate the similarity between two word vectors in their vector representations using the following formula:

[0158]

[0159] Among them, v i and v j This represents two nodes in the same node pair. Represents node v i Word vectors, Represents node v j The word vector, p(v i ,v j ) represents a node pair (v i ,v j The similarity in vector representation between the word vectors corresponding to the two nodes in the equation.

[0160] Optionally, the loss calculation submodule is used to calculate the loss value using the following objective function:

[0161]

[0162] Where J represents the objective function, w ij Represents a pair of nodes (v) i ,v j The cosine similarity between the word vectors corresponding to the two nodes in the equation is denoted by log, where log represents the logarithmic function and λ is the regularization factor.

[0163] Optionally, the text vector iteration module 402 is used for:

[0164] Based on the vector dimensions of the real word vectors, generate the text vectors of the same dimensions.

[0165] Figure 6 A block diagram of a word representation apparatus in an exemplary embodiment is shown. Please refer to... Figure 6 The word representation device 500 includes:

[0166] The second word segmentation module 501 is used to acquire the target text and segment the target text to obtain multiple words;

[0167] The graph generation module 502 is used to treat each word as a node and connect the nodes corresponding to adjacent words in the target text to generate an undirected graph.

[0168] The random walk module 503 is used to take each node as a starting node and perform a random walk on the undirected graph starting from the starting node to obtain a set of node sequences corresponding to each starting node, and obtain multiple sets of node sequences in total.

[0169] The node combination module 504 is used to combine the nodes in each node sequence in pairs in order to obtain multiple node pairs.

[0170] The word vector iteration module 505 is used to generate word vectors corresponding to each node, and uses the similarity of the word vectors corresponding to two nodes in the same node pair in the vector representation as the optimization objective to iteratively optimize the word vectors corresponding to each node.

[0171] The word vector acquisition module 506 is used to obtain the final word vector representing each word after the iterative optimization is completed.

[0172] Optionally, the word vector iteration module 505 includes:

[0173] The similarity calculation module is used to calculate the similarity between two word vectors in vector representation based on the word vectors corresponding to the two nodes in each node pair in each iteration;

[0174] The loss calculation module is used to calculate the loss value based on the similarity corresponding to each node.

[0175] The word vector update module is used to update the word vector corresponding to each node by gradient descent based on the loss value.

[0176] Optionally, the similarity calculation module is used to calculate the similarity between two word vectors in their vector representations using the following formula:

[0177]

[0178] Among them, v i and v j This represents two nodes in the same node pair. Represents node v i Word vectors, Represents node v j The word vector, p(v i ,v j ) represents a node pair (v i ,v j The similarity in vector representation between the word vectors corresponding to the two nodes in the equation.

[0179] Optionally, the loss calculation module is used to calculate the loss value using the following objective function:

[0180]

[0181] Where J represents the objective function, w ij Represents a pair of nodes (v) i ,v j The cosine similarity between the word vectors corresponding to the two nodes in the equation is denoted by log, where log represents the logarithmic function and λ is the regularization factor.

[0182] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0183] Figure 7 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0184] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the aforementioned word representation method or text representation method. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0185] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the word representation method or text representation method described above.

[0186] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the word representation method or text representation method described above. For example, the computer-readable storage medium may be the memory 702 including the program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the word representation method or text representation method described above.

[0187] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described word representation method or text representation method when executed by the programmable device.

[0188] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0189] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0190] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method of representing text, characterized by, The method comprises the following steps: obtaining a first target text and performing word segmentation on the first target text to obtain a word sequence; generating a text vector of the first target text and performing at least one iteration on the text vector, each iteration comprising multiple sub-iterations; in the i-th sub-iteration of each iteration, based on a sliding window with a length of k, after sliding the sliding window of the last sub-iteration on the word sequence backward, combining the real word vector of the k words in the current sliding window and the text vector optimized in the last sub-iteration by weighting, and taking the combined vector as the predicted word vector of the next word outside the current sliding window; optimizing the text vector according to the predicted word vector and the real word vector of the next word; after completing the at least one iteration, obtaining a final text vector for representing the first target text; the real word vector of each word is obtained by the following method: obtaining a second target text and performing word segmentation on the second target text to obtain multiple words; taking each word as a node respectively, and connecting the nodes corresponding to the words appearing adjacently in the second target text to generate an undirected graph; taking each node as a starting node respectively, and performing random walk on the undirected graph starting from the starting node to obtain a group of node sequences corresponding to each starting node, a total of multiple groups of node sequences; combining the nodes in each group of node sequences in order two by two to obtain multiple node pairs; generating a word vector corresponding to each node pair, and iteratively optimizing the word vector corresponding to each node pair by taking the similarity of the word vectors corresponding to the two nodes in the same node pair in the vector representation as an optimization target; after the iterative optimization ends, obtaining a final word vector for representing each word, and taking the final word vector of each word as the real word vector.

2. The method of claim 1, wherein, the iterative optimization of the word vector corresponding to each node pair by taking the similarity of the word vectors corresponding to the two nodes in the same node pair in the vector representation as an optimization target comprises: in each iteration, calculating the similarity of the two word vectors in the vector representation according to the word vectors corresponding to the two nodes in each node pair respectively; calculating a loss value according to the similarity corresponding to each node pair; updating the word vector corresponding to each node pair by gradient descent according to the loss value.

3. The method of claim 2, wherein, the similarity of the two word vectors in the vector representation is calculated by the following formula: ； wherein, v i and v j denote two nodes in the same node pair, denote a node v i word vector of denote a node v j word vector of p ( v i , v j ) denote the similarity of the word vectors corresponding to the two nodes in the node pair ( v i , v j ) on the vector representation.

4. The method of claim 3, wherein, the loss value is calculated by the following objective function: ； wherein, J represents the objective function, w ij represents the cosine similarity between the word vectors corresponding to the two nodes in the node pair v i , v j ) respectively, and log represents the logarithmic function, and λ is a regularization factor.

5. The method of claim 1, wherein, the generation of the text vector of the first target text comprises: generating the text vector with the same dimension as the real word vector according to the vector dimension of the real word vector.

6. A word representation method, characterized by, The method comprises the following steps: obtaining a target text and performing word segmentation on the target text to obtain multiple words; taking each word as a node respectively, and connecting the nodes corresponding to the words appearing adjacently in the target text to generate an undirected graph; taking each node as a starting node respectively, and performing random walk on the undirected graph starting from the starting node to obtain a group of node sequences corresponding to each starting node, a total of multiple groups of node sequences; combining the nodes in each group of node sequences in order two by two to obtain multiple node pairs; The word vector corresponding to each node is generated, and similarity of the word vectors corresponding to two nodes in the same node pair in vector representation is taken as an optimization target to iteratively optimize the word vector corresponding to each node pair; After the iterative optimization ends, a final word vector used for representing each word is obtained.

7. A text representation apparatus characterized by comprising: The method comprises the following steps: The first tokenization module is configured to obtain a first target text and perform tokenization on the first target text to obtain a word sequence; The text vector iteration module is configured to generate a text vector of the first target text and perform at least one round of iteration on the text vector, each round of iteration comprising multiple sub-iterations; in the i th sub-iteration of each round of iteration, based on a sliding window with a length of k, after the sliding window of the last sub-iteration on the word sequence is slid backward, the real word vector of the k words in the current sliding window and the text vector optimized in the last sub-iteration are combined by weighting, the vector after the weighting combination is taken as a predicted word vector of a next word outside the current sliding window, and the text vector is optimized according to the predicted word vector and the real word vector of the next word; The text vector obtaining module is configured to obtain a final text vector used for representing the first target text after the at least one round of iteration is completed; The real word vector of each word is obtained by the following modules: The tokenization submodule is configured to obtain a second target text and perform tokenization on the second target text to obtain multiple words; The graph generation submodule is configured to take each word as a node and connect nodes corresponding to adjacent words in the second target text to generate an undirected graph; The random walk submodule is configured to take each node as a starting node and perform random walk on the undirected graph starting from the starting node to obtain a group of node sequences corresponding to each starting node, and a total of multiple groups of node sequences are obtained; The node combination submodule is configured to combine nodes in each group of node sequences in order two by two to obtain multiple node pairs; The word vector iteration submodule is configured to generate a word vector corresponding to each node pair and take similarity of the word vectors corresponding to two nodes in the same node pair in vector representation as an optimization target to iteratively optimize the word vector corresponding to each node pair; The word vector obtaining submodule is configured to obtain a final word vector used for representing each word after the iterative optimization ends, and take the final word vector of each word as the real word vector.

8. A word representation apparatus characterized by comprising: The method comprises the following steps: The second tokenization module is configured to obtain a target text and perform tokenization on the target text to obtain multiple words; The graph generation module is configured to take each word as a node and connect nodes corresponding to adjacent words in the target text to generate an undirected graph; The random walk module is configured to take each node as a starting node and perform random walk on the undirected graph starting from the starting node to obtain a group of node sequences corresponding to each starting node, and a total of multiple groups of node sequences are obtained; The node combination module is configured to combine nodes in each group of node sequences in order two by two to obtain multiple node pairs; The word vector iteration module is configured to generate a word vector corresponding to each node and to iteratively optimize the word vector corresponding to each node by taking the similarity of the word vectors corresponding to two nodes in the same node pair in the vector representation as an optimization target. The word vector obtaining module is configured to obtain a final word vector for representing each word after the iteration optimization is completed.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method in any one of claims 1-6.

10. An electronic device, comprising: The program is executed by the processor to implement the steps of the method in any one of claims 1-6. The program is executed by the processor to implement the steps of the method in any one of claims 1-6. The program is executed by the processor to implement the steps of the method in any one of claims 1-6. The program is executed by the processor to implement the steps of the method in any one of claims 1-6.

Images