Method and system for dynamically generating long tail words
Dynamically generate long-tail words through the Word2Vec model, solving the problems of high labor costs and insufficient dynamics in traditional methods, and achieving high-quality and dynamic long-tail words generation to meet user needs.
Patent Information
- Application Number
- CN202411210082.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-17
AI Technical Summary
The traditional long-tail word generation method relies on manual experience and static vocabulary, resulting in high labor costs and insufficient dynamics, and is unable to quickly respond to changes in real-time user needs and updates on hot topics.
Using a dynamic generation method based on Word2Vec, a high-quality long-tail word list is generated by collecting and preprocessing search word data, a word embedding model is constructed in the full historical quantity and unit time, similar word matching and verification are performed, and a high-quality long-tail word list is generated.
It reduces labor costs, improves the dynamicity and relevance of long-tail words, can better meet the needs of users and systems, and provide the latest and most relevant content.
Smart Images
Figure CN119150855A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer data processing, and in particular to a method and system for dynamically generating long-tail words. Background Art
[0002] With the rapid development of the Internet and the continuous advancement of information technology, search engines and recommendation systems have been widely used in various applications. In order to improve user experience, search engines and recommendation systems need to provide more accurate and personalized content to meet user needs. As an important part of search engine optimization and content recommendation, long-tail keywords have lower search volume but higher conversion rate and more accurate user positioning capabilities.
[0003] Traditional long-tail word generation methods usually rely on manual experience and static word libraries. This method has the following main problems:
[0004] 1. High cost of manual generation: Manually defining and maintaining a list of long-tail words requires a lot of manpower and time, and it is difficult to cover all possible long-tail words.
[0005] 2. Lack of dynamism: Static vocabulary cannot quickly respond to real-time changes in user needs and updates to hot topics, and thus cannot provide users with the latest and most relevant content. Summary of the invention
[0006] In view of this, the present invention provides a method and system for dynamically generating long-tail words to solve the above problems.
[0007] In order to solve the above technical problems, the present invention provides a method for dynamically generating long-tail words, comprising:
[0008] S1, collect search terms in a preset field and relevant text data in the field, and pre-process the collected text;
[0009] S2. Build a historical full-word embedding model and a single search word embedding model per unit time;
[0010] S3, matching similar words to the search words to form similar word roots;
[0011] S4: Perform similarity verification on the long-tail word roots generated in S3 above, and output the long-tail word roots based on the verification result.
[0012] As an optional method, in the above S1, the preprocessing includes: removing emoticons, special characters, irrelevant characters within preset rules, English character strings, line breaks and blank characters in the text.
[0013] As an optional method, in the above S2, a historical full-quantity word embedding model is constructed, including the following method:
[0014] After completing the preprocessing of the text, a vocabulary is constructed, which is used to index and vectorize the words in subsequent model training;
[0015] Use the gensim library to define the architecture of the Skip-gram model and import the constructed vocabulary into the word2vec model of the gensim library for training;
[0016] After the training is completed, save the trained word embedding model to a file;
[0017] Calculate the cosine similarity between word vectors and randomly sample to verify the effect of the trained word embedding model;
[0018] After the verification is completed, a multi-dimensional word2vec word embedding model based on historical full data training is constructed as the semantic basis for subsequent long-tail word generation.
[0019] As an option, build a vocabulary that includes:
[0020] Use the word segmentation tool to load the proper nouns in the preset field, use the TestRank algorithm to extract all independent words from the preprocessed text data, and build a vocabulary.
[0021] As an optional method, in the above S2, constructing a single search term embedding model per unit time includes the following methods:
[0022] Collect and count articles published for each search term within a preset time, and build a vocabulary after preprocessing the text. The vocabulary is used to index and vectorize words in subsequent model training.
[0023] Use the gensim library to define the architecture of the Skip-gram model and import the constructed vocabulary into the word2vec model of the gensim library for training;
[0024] After the training is completed, save the trained word embedding model to a file;
[0025] Calculate the cosine similarity between word vectors and randomly sample to verify the effect of the trained word embedding model;
[0026] After the verification is completed, a multi-dimensional word2vec word embedding model based on single search word training per unit time is constructed as the semantic basis for subsequent long-tail word generation;
[0027] The vocabulary is constructed by using a general word segmentation operation to extract independent words that meet preset requirements.
[0028] As an optional method, in the above S3, similar word matching of the search word is to use the cosine similarity formula to calculate the similarity between the target word and other words in the vocabulary, and measure the similarity of the two vectors by calculating the dot product and the norm.
[0029] As an alternative, form similar word roots, including:
[0030] The historical full-word embedding model is used to match words similar to the search term as the first root word, and the single search term embedding model in unit time is used to match words similar to the search term as the second root word. The first root word and the second root word are concatenated in order to form a similar word root word.
[0031] As an optional method, in the above S4, the similarity check of the long-tail word root is performed, including:
[0032] Segment the proper nouns in the preset field and generate a vocabulary of independent words;
[0033] Use the TfidfVectorizer method of the scikit-learn library to calculate the TF-IDF value of the vocabulary;
[0034] Segment the word roots and calculate their TF-IDF values;
[0035] Based on the word segmentation results of the root, check whether the TF-IDF value of each word is greater than the preset threshold. If so, it is considered that the root meets the requirements.
[0036] As an optional method, in the above S4, long-tail words are generated according to the post type, including:
[0037] According to the verified word roots, obtain the classification results of recent search word posts, and generate the final long-tail words based on the classification results and the classification result mapping words.
[0038] On the other hand, the present invention also provides a system for dynamically generating long-tail words, comprising:
[0039] A data preprocessing module, which is used to collect search terms in a preset field and relevant text data in the field, and preprocess the collected text;
[0040] Model building module: The model building module is used to build a historical full-word embedding model and a single search word embedding model per unit time;
[0041] Matching module, the matching module is used to match similar words to the search words and form similar word roots;
[0042] The root generation module is used to perform similarity verification on the long-tail word roots generated by S3 above, and output the long-tail word roots according to the verification results.
[0043] The beneficial effects of the present invention are:
[0044] The present invention proposes a method and system for dynamically generating long-tail words based on Word2Vec, which can efficiently generate a high-quality long-tail word list by constructing and training a Word2Vec model suitable for a specific application field. This method can not only reduce labor costs, but also significantly improve the dynamics and relevance of generating long-tail words, and better meet the needs of users and systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A schematic diagram of a process for dynamically generating long-tail words provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0046] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods.
[0047] See also Figure 1 This embodiment provides a method and system for dynamically generating long-tail words based on Word2Vec. By building and training a Word2Vec model suitable for a specific application field, a high-quality long-tail word list can be efficiently generated. This method can not only reduce labor costs, but also significantly improve the dynamics and relevance of generating long-tail words, better meeting the needs of users and systems.
[0048] This embodiment is divided into 4 steps in total:
[0049] S1, collect search terms in a preset field and relevant text data in the field, and pre-process the collected text;
[0050] S2. Build a historical full-word embedding model and a single search word embedding model per unit time;
[0051] S3, matching similar words to the search words to form similar word roots;
[0052] S4: Perform similarity verification on the long-tail word roots generated in S3 above, and output the long-tail word roots based on the verification result.
[0053] Among them, in S1, the search terms in a specific field and the real-time relevant text data are collected. The relevant search terms in a specific field (such as medical aesthetics) are collected, such as terms like double eyelid, rhinoplasty, chin augmentation, whitening, anti-aging, etc. According to the collected phrases, relevant articles are collected or crawled (such as on the Weibo platform). The preprocessing includes: removing emoticons, special characters, irrelevant characters within the preset rules (such as @1233, #hello#), English strings, line breaks, and whitespace characters from the text.
[0054] In an optional implementation scenario, the historical full-scale data (excluding the current year) includes 360,785 medical aesthetics articles on the Xiaohongshu platform and 1,745,309 medical aesthetics articles on the Weibo platform. Based on this, S2 adopts the following method:
[0055] After completing the text preprocessing of the historical full-scale data, the construction of the historical full-scale word embedding model begins.
[0056] The construction steps are as follows:
[0057] 1. Build a vocabulary
[0058] When constructing the historical full-scale word embedding model, a word segmentation tool is needed to load the proper nouns in a specific field (such as terms like hyaluronic acid injection for eyebrow augmentation). All independent words are extracted from the preprocessed text data using the TestRank algorithm, and a vocabulary is constructed. This vocabulary will be used for the indexing and vectorization of words in subsequent model training.
[0059] 2. Model definition
[0060] Use the gensim library to define the architecture of a Skip-gram model. Skip-gram is one of the two training methods of word2vec, which can better capture the relevance of words in the context during training.
[0061] 3. Model training
[0062] Import the constructed vocabulary into the word2vec model of the gensim library for training. Optionally, this embodiment provides the following model parameters:
[0063] vector_size = 300: The dimension of the trained word vector is 300 dimensions;
[0064] window = 5: The size of the context window sliding;
[0065] min_count = 5: Words with a word frequency less than 5 will be ignored;
[0066] workers = 16: Use 16 CPU threads for parallel training.
[0067] 4. Save the model
[0068] After the training is completed, the trained word embedding model is saved to a file for subsequent use and loading.
[0069] 5. Model Validation
[0070] By calculating the cosine similarity between word vectors, random sampling is used to verify the effect of the trained word embedding model to ensure that it can correctly capture the semantic relationship between words.
[0071] Through the above steps, a 300-dimensional word2vec word embedding model trained based on the full amount of historical data is constructed, providing a reliable semantic basis for the subsequent long-tail word generation.
[0072] In addition, to construct a single search term embedding model within a unit time, we need to count the articles published for each search term within a preset time (such as the past 7 days), and then build a corresponding word embedding model based on each search term. Its construction method is similar to the historical full word embedding model, including:
[0073] Collect and count articles published for each search term within a preset time, and build a vocabulary after preprocessing the text. The vocabulary is used to index and vectorize words in subsequent model training.
[0074] Use the gensim library to define the architecture of the Skip-gram model and import the constructed vocabulary into the word2vec model of the gensim library for training;
[0075] After the training is completed, save the trained word embedding model to a file;
[0076] Calculate the cosine similarity between word vectors and randomly sample to verify the effect of the trained word embedding model;
[0077] After the verification is completed, a 300-dimensional word2vec word embedding model based on single search word training per unit time is constructed as the semantic basis for subsequent long-tail word generation;
[0078] The vocabulary is constructed by using a general word segmentation operation to extract independent words that meet preset requirements.
[0079] Afterwards, in S3, based on the output results of the above two models, the cosine similarity formula is used to calculate the similarity between the target word and other words in the vocabulary, and the similarity of the two vectors is measured by calculating the dot product and norm. The formula for calculating cosine similarity is as follows:
[0080]
[0081] in:
[0082] 1) A and B are two word vectors;
[0083] 2) A*B is the dot product of vectors A and B, that is, the sum of the products of the elements of the corresponding dimensions;
[0084] 3) |A| and |B| are the norms (i.e., the lengths of the vectors) of vectors A and B, respectively, and are calculated as the square root of the sum of the squares of all dimensional elements.
[0085] The word vector model constructed based on the above S2 matches words similar to the search term, and uses the word as the first root word (such as the similar words for double eyelids are swollen eyes and double eyelids); then, based on step 4, the word vector model is constructed to match recent similar words as the second root word (such as the similar words for treatment are repair and expert), and finally form the root word of the long-tail word (such as swollen eyes and double eyelid expert or swollen eyes and double eyelid repair expert).
[0086] After completing the generation of similar word roots through the above scheme, it is also necessary to verify the similar word roots:
[0087] Check the domain of the output word roots to see if they meet the requirements. Use the word segmentation and TF-IDF-based methods to verify. The method is as follows:
[0088] Segment the proper nouns in a specific field and generate a vocabulary of independent words;
[0089] Use the TfidfVectorizer method of the scikit-learn library to calculate the TF-IDF value of the vocabulary;
[0090] Segment the word roots and calculate their TF-IDF values;
[0091] Based on the word segmentation results of the root, check whether the TF-IDF value of each word is greater than the set threshold (such as 0.9). If so, it is considered that the root meets the requirements.
[0092] In an optional scenario, for example, the test result of the root word "puffy eye double eyelid repair" has a threshold greater than 0.9, so it is a valid root word. The test result of the root word "double eyelid lactic acid nasolabial fold repair" has a threshold less than 0.9, so it is an invalid root word.
[0093] Finally, based on the verified word roots, the classification results of recent search word posts are obtained, and the final long-tail words are generated based on the classification results and the word mapping results. Based on the generated long-tail words, you can use the big model to create related long-tail word articles and explore more content creation opportunities in different segments. This kind of diverse creation can not only meet the needs of a wider range of users, but also create a more colorful content ecosystem. At the same time, this diversity also helps to increase the depth and breadth of the content and improve the retrieval weight in the search engines of social platforms.
[0094] This embodiment is based on the method and system of dynamically generating long-tail words based on Word2Vec. By constructing and training a Word2Vec model suitable for a specific application field, a high-quality long-tail word list can be efficiently generated. This method can not only reduce labor costs, but also significantly improve the dynamics and relevance of generating long-tail words, better meeting the needs of users and systems. And, according to the scheme described in this embodiment, it is possible to reduce labor costs and significantly improve the dynamics and relevance of generating long-tail words; it is highly flexible, and the word roots can be maintained and dynamically updated according to usage requirements; it is highly extensible, and users only need to provide field search terms without having to master complex editing skills to obtain the corresponding long-tail words; it is highly timely, and according to the article data collected in real time, the article release time can be flexibly selected for long-tail word mining.
[0095] In addition, this embodiment also provides a system for dynamically generating long-tail words, as an optional method, including:
[0096] A data preprocessing module, which is used to collect search terms in a preset field and relevant text data in the field, and preprocess the collected text;
[0097] Model building module: The model building module is used to build a historical full-word embedding model and a single search word embedding model per unit time;
[0098] Matching module, the matching module is used to match similar words to the search words and form similar word roots;
[0099] The root generation module is used to perform similarity verification on long-tail word roots and output the long-tail word roots based on the verification results.
[0100] The above are only preferred embodiments of the present invention. It should be noted that the above preferred embodiments should not be regarded as limiting the present invention, and the protection scope of the present invention should be based on the scope defined by the claims. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the spirit and scope of the present invention, and these improvements and modifications should also be regarded as the protection scope of the present invention.
Claims
1. A method for dynamically generating long-tail words, characterized in that: include: S1, collect search terms in a preset field and relevant text data in the field, and pre-process the collected text; S2. Build a historical full-word embedding model and a single search word embedding model per unit time; S3, matching similar words to the search words to generate long-tail word roots; S4: Perform similarity verification on the long-tail word roots generated in S3 above, and output the long-tail word roots based on the verification result.
2. A method for dynamically generating long-tail words according to claim 1, characterized in that: In the above S1, the preprocessing includes: removing emoticons, special characters, irrelevant characters within preset rules, English character strings, line breaks and blank characters in the text.
3. A method for dynamically generating long-tail words according to claim 1, characterized in that: In the above S2, the construction of the historical full-quantity word embedding model includes the following method: After completing the preprocessing of the text, a vocabulary is constructed, which is used to index and vectorize the words in subsequent model training; Use the gensim library to define the architecture of the Skip-gram model and import the constructed vocabulary into the word2vec model of the gensim library for training; After the training is completed, save the trained word embedding model to a file; Calculate the cosine similarity between word vectors and randomly sample to verify the effect of the trained word embedding model; After the verification is completed, a multi-dimensional word2vec word embedding model based on historical full data training is constructed as the semantic basis for subsequent long-tail word generation.
4. A method for dynamically generating long-tail words according to claim 3, characterized in that: The building of the vocabulary includes: Use the word segmentation tool to load the proper nouns in the preset field, use the TestRank algorithm to extract all independent words from the preprocessed text data, and build a vocabulary.
5. The method for dynamically generating long-tail words according to claim 1, characterized in that: In the above S2, the construction of a single search term embedding model per unit time includes the following methods: Collect and count articles published for each search term within a preset time, and build a vocabulary after preprocessing the text. The vocabulary is used to index and vectorize words in subsequent model training. Use the gensim library to define the architecture of the Skip-gram model and import the constructed vocabulary into the word2vec model of the gensim library for training; After the training is completed, save the trained word embedding model to a file; Calculate the cosine similarity between word vectors and randomly sample to verify the effect of the trained word embedding model; After the verification is completed, a multi-dimensional word2vec word embedding model based on single search word training per unit time is constructed as the semantic basis for subsequent long-tail word generation; The word list construction is to use a common word segmentation operation to extract independent words that meet preset requirements.
6. A method for dynamically generating long-tail words according to claim 3 or 5, characterized in that: In the above S3, similar word matching of the search word is to use the cosine similarity formula to calculate the similarity between the target word and other words in the vocabulary, and the similarity of the two vectors is measured by calculating the dot product and the norm.
7. A method for dynamically generating long-tail words according to claim 6, characterized in that: Generate long-tail word roots, including: The historical full-word embedding model is used to match words similar to the search term as the first root, and the single search term embedding model in unit time is used to match words similar to the search term as the second root. The first root and the second root are concatenated in sequence to form a long-tail word root.
8. A method for dynamically generating long-tail words according to claim 7, characterized in that: In the above S4, the similarity check of the long-tail word root includes: Segment the proper nouns in the preset field and generate a vocabulary of independent words; Use the TfidfVectorizer method of the scikit-learn library to calculate the TF-IDF value of the vocabulary; Segment the word roots and calculate their TF-IDF values; Based on the word segmentation results of the root, check whether the TF-IDF value of each word is greater than the preset threshold. If so, it is considered that the root meets the requirements.
9. The method for dynamically generating long-tail words according to claim 7, characterized in that: In the above S4, the outputting of long-tail word roots includes: According to the verified word roots, obtain the classification results of recent search word posts, and generate the final long-tail words based on the classification results and the classification result mapping words.
10. A system for dynamically generating long-tail words, characterized in that: include: A data preprocessing module, which is used to collect search terms in a preset field and relevant text data in the field, and preprocess the collected text; A model building module, which is used to build a historical full-word embedding model and a single search word embedding model per unit time; A matching module, wherein the matching module is used to match similar words to the search words to form similar word roots; The root generation module is used to perform similarity verification on long-tail word roots and output the long-tail word roots according to the verification result.