Text processing method and device based on Node2Vec, storage medium and electronic equipment

Abstract

The application relates to the field of natural language processing, and discloses a text processing method and device based on Node2Vec, a storage medium and electronic equipment. The application obtains a word set and a character set by respectively performing word segmentation processing and single-character segmentation processing on target text; calculates TF-IDF values of the words and the characters; respectively performs sliding window operation on the words and the characters to obtain bigram words and bigram characters; constructs tri-gram words and tri-gram characters according to the bigram words and the bigram characters, and takes the TF-IDF values as weights; inputs the tri-gram words and the tri-gram characters into a Node2Vec model for training, and the Node2Vec model outputs word node vectors and character node vectors; and constructs a text vectorization representation according to the word node vectors and the character node vectors. The application reduces the influence of word segmentation on the result, improves the accuracy of text vector representation, and more comprehensively obtains a text semantic representation.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and more specifically, to a text processing method, apparatus, storage medium, and electronic device based on Node2Vec. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. Natural language processing (NLP) is an important direction within computer science and AI, researching various theories and methods that enable effective communication between humans and computers using natural language. Predicting key phrases from text is a crucial research area in NLP. Text vectorization refers to the process of converting natural language text into digital vector representations that machines can understand; it is a primary method of NLP.

[0003] There is a text representation processing method based on the bag-of-words model. It performs word segmentation, pause word removal, low-frequency word removal, and feature selection on the collected text dataset, and then uses a spatial vector model to represent the processed text. At the same time, it uses a neural network method to train word vectors on the processed text. Based on the similarity of the word vectors, it modifies the weights of the feature words in the bag-of-words model to obtain a new text representation model.

[0004] However, the above methods only perform word-level segmentation and word-level vectorization of the text, and cannot obtain the global relationships between characters or words, which will reduce the accuracy of the text vector representation. Summary of the Invention

[0005] To improve the accuracy of text vector representation, this invention provides a text processing method, apparatus, storage medium, and electronic device based on Node2Vec.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] Firstly, this invention proposes a text processing method based on Node2Vec, comprising:

[0008] S1: Get the text.

[0009] S2: Perform word segmentation and single-character segmentation on the text to obtain a word set and a character set.

[0010] S3: Calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set.

[0011] S4: Perform a sliding window operation on each word in the word set and each character in the character set to obtain binary words and binary characters.

[0012] S5: Based on the bigram words and bigram characters, construct trigram words and trigram characters by weighting the TF-IDF value of each word and the TF-IDF value of each character, respectively.

[0013] S6: Input the triplet words and triplet characters into the Node2Vec model for training. The Node2Vec model outputs word node vectors and byte node vectors.

[0014] S7: Construct a text vectorized representation based on the word node vector and the byte node vector.

[0015] As a preferred embodiment, the text vectorization representation is constructed based on the word node vector and the byte node vector, and its expression is as follows:

[0016]

[0017]

[0018] S = [S w ,S c ]

[0019] Among them, S word Let E represent the word-level vector of the text, where i represents the tag of the first word of the text, j represents the tag of the last word of the text, and E represents the word-level vector of the text. i S represents the word node vector. character This represents a word-level vector of text, where m represents the tag of the first word of the text, n represents the tag of the last word of the text, and E... n represents a byte-level vector, and S represents a text vector obtained by concatenating word-level vectors and character-level vectors.

[0020] As a preferred approach, in S4, a sliding window operation is performed on each word in the word set and each character in the character set according to the bigram.

[0021] As a preferred option, in S6, the vector dimension of the Node2Vec model is set to N dimensions.

[0022] Secondly, the present invention also proposes a text processing device based on Node2Vec, comprising:

[0023] The acquisition unit is used to acquire text.

[0024] The word segmentation unit is used to segment the text into words to obtain a word set.

[0025] The character segmentation unit is used to perform single-character segmentation on the text to obtain a character set.

[0026] The computation unit is used to calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set.

[0027] The binary tuple construction unit is used to perform sliding window operations on each word in the word set and each character in the character set to obtain binary words and binary characters.

[0028] The triplet construction unit is used to construct triplet words and triplet characters based on the bigram words and the bigram characters, respectively, using the TF-IDF value of each word and the TF-IDF value of each character as weights.

[0029] The Node2Vec model is trained by inputting the triplet words and triplet characters into the Node2Vec model, and the Node2Vec model outputs word node vectors and byte node vectors.

[0030] The text representation unit is used to construct a vectorized representation of text based on the word node vector and the byte node vector.

[0031] As a preferred embodiment, the text representation unit constructs a text vectorization representation based on the word node vector and the byte node vector, the expression of which is as follows:

[0032]

[0033]

[0034] S = [S w ,c c ]

[0035] Among them, S word Let E represent the word-level vector of the text, where i represents the tag of the first word of the text, j represents the tag of the last word of the text, and E represents the word-level vector of the text. i S represents the word node vector. character This represents a word-level vector of text, where m represents the tag of the first word of the text, n represents the tag of the last word of the text, and E... n S represents a byte-level vector, and S represents a text vector obtained by concatenating word-level vectors and character-level vectors.

[0036] As a preferred embodiment, the bigram construction unit performs sliding window operations on each word in the word set and each character in the character set according to the bigram.

[0037] As a preferred embodiment, the Node2Vec model has a vector dimension of N dimensions.

[0038] Thirdly, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the Node2Vec-based text processing method of the first aspect.

[0039] Fourthly, the present invention also proposes an electronic device including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, implements the Node2Vec-based text processing method of the first aspect.

[0040] Compared with existing technologies, the beneficial effects of the technical solution of this invention are as follows: Based on text segmentation, a word is represented by character-level n-grams through character segmentation and word segmentation. The TF-IDF value of each character and word is calculated separately for both characters and words. A sliding window operation is performed on each word and each character to build relationships between words and characters. The TF-IDF value of each word and the TF-IDF value of each character are used as weights to construct triplet words and triplet characters. Finally, character-level and word-level vector representations are obtained. By constructing a network structure and using random walks in social networks to update model parameters, text vector representations are obtained. At the character level, the problem of feature loss caused by certain word segmentation errors is compensated for. Word segmentation errors also introduce character-level features into the network, reducing the impact of word segmentation on the results, improving the accuracy of text vector representation, and obtaining a more comprehensive semantic representation of the text. Attached Figure Description

[0041] Figure 1 This is a flowchart of the text processing method based on Node2Vec in Example 1.

[0042] Figure 2 This is a flowchart of the text processing method based on Node2Vec in Example 2.

[0043] Figure 3 This is an architecture diagram of the text processing device based on Node2Vec in Example 3. Detailed Implementation

[0044] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0045] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] Example 1

[0047] Please see Figure 1 This embodiment proposes a text processing method based on Node2Vec, including:

[0048] S1: Get the text.

[0049] S2: Perform word segmentation and single-character segmentation on the text to obtain a word set and a character set.

[0050] S3: Calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set.

[0051] S4: Perform a sliding window operation on each word in the word set and each character in the character set to obtain binary words and binary characters.

[0052] S5: Based on the bigram words and bigram characters, construct trigram words and trigram characters by weighting the TF-IDF value of each word and the TF-IDF value of each character, respectively.

[0053] S6: Input the triplet words and triplet characters into the Node2Vec model for training. The Node2Vec model outputs word node vectors and byte node vectors.

[0054] S7: Construct a text vectorized representation based on the word node vector and the byte node vector.

[0055] In the specific implementation process, based on text segmentation, character-level n-grams are introduced to represent a word through character segmentation and word segmentation. The TF-IDF value of each character and word is calculated separately for both characters and words. Sliding window operation is performed on each word and each character to build relationships between words and characters. The TF-IDF value of each word and the TF-IDF value of each character are used as weights to construct triplet words and triplet characters. Finally, character-level and word-level vector representations are obtained. By constructing a network structure and using random walks in social networks to update model parameters, text vector representations are obtained. At the character level, the problem of feature loss caused by certain word segmentation errors is compensated for. Word segmentation errors are also introduced into the network at the character level, reducing the impact of word segmentation on the results, improving the accuracy of text vector representation, and obtaining a more comprehensive semantic representation of the text.

[0056] Example 2

[0057] Please see Figure 2 This embodiment proposes a text processing method based on Node2Vec, including:

[0058] S1: Get the text.

[0059] S2: Perform word segmentation and single-character segmentation on the text to obtain a word set W(w1, w2, w3, ... w n ) and word set C(c1,c2,c3,...clen(s) ).

[0060] S3: Calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set.

[0061] In this embodiment, after calculating the word frequency and character frequency, the TF-IDF value of each word and the TF-IDF value of each character are obtained by multiplying the word frequency by the inverse document coefficient and the character frequency by the inverse document coefficient, respectively.

[0062] S4: Perform a sliding window operation on each word in the word set and each character in the character set to obtain binary words and binary characters.

[0063] In this embodiment, a sliding window operation is performed on each word in the word set and each character in the character set according to the bigram. For example, the bigram constructed from the sequence of words: word1, word2, word3, word4 is (word1, word2)(word2, word3)(word3, word4).

[0064] S5: Based on the bigram words and bigram characters, construct trigram words and trigram characters by weighting the TF-IDF value of each word and the TF-IDF value of each character, respectively.

[0065] Taking words as an example, such as words1 and word2, with corresponding weights w1 and w2 respectively, the constructed triplet would be (word1, word2, w1+w2).

[0066] In the specific implementation process, a sliding window operation is performed on each word in the word set and each character in the character set according to the bigram, thereby obtaining the relationships between words and characters, such as bigrams (w i ,w i ) and binary words (c m ,c n And use the TF-IDF value of the corresponding word or character as the weight of the two, for example, w i The TF-IDF value is 0.2356, w i The TF-IDF value is 0.6018, w i With w j The weight is (0.2356 + 0.6018) / 2 = 0.4187, and similarly, we can obtain c. m With c n The weights are used to obtain the trigrams (w i ,w i weight ij ) and the word (c m ,c nweight mn ).

[0067] S6: Input the triplet words and triplet characters into the Node2Vec model for training. The Node2Vec model outputs word node vector E. i and byte point vector E n .

[0068] In this embodiment, the size parameter of the Node2Vec model is set so that the vector dimension of the Node2Vec model is N dimensions.

[0069] S7: Based on the word node vector and the byte node vector, construct the text vectorized representation, the expression of which is as follows:

[0070]

[0071]

[0072] S = [S w ,S c ]

[0073] Among them, S word Let E represent the word-level vector of the text, where i represents the tag of the first word of the text, j represents the tag of the last word of the text, and E represents the word-level vector of the text. i The word-level vector S is obtained by summing and averaging all the word-node vectors in the text. w S character This represents a word-level vector of text, where m represents the tag of the first word of the text, n represents the tag of the last word of the text, and E... n The word-level vector S is obtained by averaging the sum of all word node vectors in the text. n S represents the word-level vector S w And word level vector S n The concatenated text vector will have a word-level vector S. w And word level vector S n By concatenating the elements, we obtain a 2N-dimensional vector S = [S w ,S c ].

[0074] In the specific implementation process, based on text segmentation, character-level n-grams are introduced to represent a word through character segmentation and word segmentation. The TF-IDF value of each character and word is calculated separately for both characters and words. Sliding window operation is performed on each word and each character to build relationships between words and characters. The TF-IDF value of each word and the TF-IDF value of each character are used as weights to construct triplet words and triplet characters. Finally, character-level and word-level vector representations are obtained. By constructing a network structure and using random walks in social networks to update model parameters, text vector representations are obtained. At the character level, the problem of feature loss caused by certain word segmentation errors is compensated for. Word segmentation errors are also introduced into the network at the character level, reducing the impact of word segmentation on the results, improving the accuracy of text vector representation, and obtaining a more comprehensive semantic representation of the text.

[0075] Example 3

[0076] Please see Figure 3 This embodiment proposes a text processing device based on Node2Vec, comprising:

[0077] The acquisition unit is used to acquire text.

[0078] In this embodiment, the text representation unit constructs a text vectorization representation based on the word node vector and the byte node vector, and its expression is as follows:

[0079]

[0080]

[0081] S = [S w ,S c ]

[0082] Among them, S word Let E represent the word-level vector of the text, where i represents the tag of the first word of the text, j represents the tag of the last word of the text, and E represents the word-level vector of the text. i S represents the word node vector. character This represents a word-level vector of text, where m represents the tag of the first word of the text, n represents the tag of the last word of the text, and E... n S represents a byte-level vector, and S represents a text vector obtained by concatenating word-level vectors and character-level vectors.

[0083] The word segmentation unit is used to segment the text into words to obtain a word set.

[0084] The character segmentation unit is used to perform single-character segmentation on the text to obtain a character set.

[0085] The computation unit is used to calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set.

[0086] The binary tuple construction unit is used to perform sliding window operations on each word in the word set and each character in the character set to obtain binary words and binary characters.

[0087] In this embodiment, the bigram construction unit performs sliding window operations on each word in the word set and each character in the character set according to the bigram.

[0088] The triplet construction unit is used to construct triplet words and triplet characters based on the bigram words and the bigram characters, respectively, using the TF-IDF value of each word and the TF-IDF value of each character as weights.

[0089] The Node2Vec model is trained by inputting the triplet words and triplet characters into it, and outputs word node vectors and byte node vectors. The vector dimension of the Node2Vec model is N-dimensional.

[0090] The text representation unit is used to construct a vectorized representation of text based on the word node vector and the byte node vector.

[0091] In the specific implementation process, based on text segmentation, character-level n-grams are introduced to represent a word through character segmentation and word segmentation. The TF-IDF value of each character and word is calculated separately for both characters and words. Sliding window operation is performed on each word and each character to build relationships between words and characters. The TF-IDF value of each word and the TF-IDF value of each character are used as weights to construct triplet words and triplet characters. Finally, character-level and word-level vector representations are obtained. By constructing a network structure and using random walks in social networks to update model parameters, text vector representations are obtained. At the character level, the problem of feature loss caused by certain word segmentation errors is compensated for. Word segmentation errors are also introduced into the network at the character level, reducing the impact of word segmentation on the results, improving the accuracy of text vector representation, and obtaining a more comprehensive semantic representation of the text.

[0092] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0093] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A text processing method based on Node2Vec, characterized in that, include: S1: Get the text; S2: Perform word segmentation and single-character segmentation on the text to obtain a word set and a character set; S3: Calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set; S4: Perform a sliding window operation on each word in the word set and each character in the character set to obtain the word pair and the character pair; S5: Based on the bigram words and bigram characters, construct triplet words and triplet characters by weighting the TF-IDF value of each word and the TF-IDF value of each character, respectively. Taking words as an example, such as words1 and word2, with corresponding weights w1 and w2 respectively, the constructed triple is (word1, word2, w1+w2). By performing a sliding window operation on each word in the word set and each character in the character set using bigrams, the relationships between words and characters are obtained sequentially, such as bigrams. and binary words And the TF-IDF value of the corresponding word or character is used as the weight of the two, and similarly, the results are obtained. and The weights are used to obtain the trigrams. and words ; S6: Input the triplet words and triplet characters into the Node2Vec model for training. The Node2Vec model outputs word node vectors and byte node vectors. S7: Construct a text vectorized representation based on the word node vector and the byte node vector; in S7, the expression for constructing the text vectorized representation based on the word node vector and the byte node vector is as follows: ； in, A word-level vector representing the text. i A marker indicating the first word of the text. j A marker indicating the last word of the text. Represents word node vectors, A word-level vector representing text. m A marker indicating the first word of the text. n A marker indicating the last word of the text. Represents a byte point vector. S This represents a text vector obtained by concatenating word-level vectors and character-level vectors. In S4, a sliding window operation is performed on each word in the word set and each character in the character set according to the bigram.

2. The text processing method based on Node2Vec according to claim 1, characterized in that, In S6, the vector dimension of the Node2Vec model is set to N dimensions.

3. A text processing device based on Node2Vec, characterized in that, include: The acquisition unit is used to acquire text; The word segmentation unit is used to segment the text into words to obtain a word set; The character segmentation unit is used to perform single-character segmentation on the text to obtain a character set; The computational unit is used to calculate the TF-IDF value of each word in the word set and the TF-IDF value of each character in the character set; The binary tuple construction unit is used to perform sliding window operations on each word in the word set and each character in the character set to obtain binary words and binary characters. The triplet construction unit is used to construct triplet words and triplet characters based on the bigram words and the bigram characters, respectively, using the TF-IDF value of each word and the TF-IDF value of each character as weights. Taking words as an example, such as words1 and word2, with corresponding weights w1 and w2 respectively, the constructed triple is (word1, word2, w1+w2). By performing a sliding window operation on each word in the word set and each character in the character set using bigrams, the relationships between words and characters are obtained sequentially, such as bigrams. and binary words And the TF-IDF value of the corresponding word or character is used as the weight of the two, and similarly, the results are obtained. and The weights are used to obtain the trigrams. and words ; The Node2Vec model is trained by inputting the triplet words and triplet characters into the Node2Vec model, and the Node2Vec model outputs word node vectors and byte node vectors. A text representation unit is configured to construct a text vectorization representation based on the word node vector and the byte node vector; the text representation unit constructs a text vectorization representation based on the word node vector and the byte node vector, and its expression is as follows: ； in, A word-level vector representing the text. i A marker indicating the first word of the text. j A marker indicating the last word of the text. Represents word node vectors, A word-level vector representing text. m A marker indicating the first word of the text. n A marker indicating the last word of the text. Represents a byte point vector. S This represents a text vector obtained by concatenating word-level vectors and character-level vectors; the bigram construction unit performs a sliding window operation on each word in the word set and each character in the character set according to the bigram.

4. The text processing device based on Node2Vec according to claim 3, characterized in that, The Node2Vec model has a vector dimension of N dimensions.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 2.

6. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the computer program is executed by the processor, it implements the method of any one of claims 1 to 2.

Images