Keyword expansion method and device based on pre-training model

Abstract

The embodiment of the present invention provides a keyword expansion method, device and electronic device based on a pre-training model, which belongs to the field of data processing technology. The method comprises: using a stop word set S to perform a stop word removal operation on each word w in a vocabulary set W to obtain a second vocabulary set; performing part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P; mapping the second vocabulary set W' keyword k into a semantic vector V through a pre-training model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W'; calculate the similarity between each keyword k and other words in the second vocabulary set W' to obtain the similarity vector S; based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S, generate the expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T. The present invention improves the accuracy of keyword expansion.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a keyword expansion method, device and electronic equipment based on a pre-training model. Background Art

[0002] Keyword Expansion is a natural language processing (NLP) technology that aims to expand the search scope or content coverage by generating other words related to a given keyword. The goal of keyword expansion is to find more relevant words without changing the original meaning in order to capture information more comprehensively. This technology has a wide range of applications in many fields, including but not limited to search engine optimization (SEO), information retrieval, text mining, question-answering systems, recommendation systems, etc.

[0003] The existing technology has the following problems in the process of implementing keyword expansion:

[0004] 1. Insufficient semantic understanding ability: Traditional keyword expansion methods mainly rely on dictionaries, synonym libraries or statistical co-occurrence analysis, which makes it difficult to capture the deep semantic relationship between words.

[0005] 2. Single expansion results: Traditional methods can usually only generate words that are closely related to the input keywords, lack diversity, and it is difficult to generate keywords that meet the needs of various contexts.

[0006] 3. Lack of context: Many keyword expansion methods ignore the meaning of keywords in specific contexts and are unable to generate keywords for specific scenarios or industries.

[0007] 4. Poor real-time performance: Traditional keyword expansion methods often rely on pre-built static vocabulary, which cannot reflect the changing trends of language in a timely manner and are difficult to meet real-time expansion needs.

[0008] The above problems become technical problems that need to be solved. Summary of the invention

[0009] In view of this, the embodiments of the present invention provide a keyword expansion method, device and electronic device based on a pre-training model, which at least partially solve the problems existing in the prior art.

[0010] In a first aspect, an embodiment of the present invention provides a keyword expansion method based on a pre-training model, comprising:

[0011] Perform a word segmentation operation on the input text T to obtain the first vocabulary set W, and use the stop word set S to perform a stop word removal operation on each word w in the vocabulary set W to obtain the second vocabulary set W'={w∈W|w S};

[0012] Perform part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P = {(w, p) | w∈W', p∈POS}, where POS represents a set of part-of-speech tags. Calculate the context of each word in the second vocabulary set W' to obtain a context set C = Ctx(T, k). The Ctx function is used to extract context information related to keyword k;

[0013] The second vocabulary set W' keyword k is mapped to a semantic vector V through the pre-trained model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W';

[0014] Calculate the similarity between each keyword k and other words in the second vocabulary set W' to obtain a similarity vector S;

[0015] Based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S, an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T is generated.

[0016] According to a specific implementation of an embodiment of the present invention, the word segmentation operation is performed on the input text T to obtain a first vocabulary set W, and a stop word set S is used to perform a stop word removal operation on each word w in the vocabulary set W to obtain a second vocabulary set W', including:

[0017] After removing special characters from the input text T, the language type of the input text T is detected;

[0018] Obtaining n word segmentation dictionaries corresponding to the language type, and generating n different word segmentation results based on the n word segmentation dictionaries;

[0019] The n different word segmentation results are removed and merged to obtain the first vocabulary set W.

[0020] According to a specific implementation of an embodiment of the present invention, the word segmentation operation is performed on the input text T to obtain a first vocabulary set W, and a stop word removal operation is performed on each word w in the vocabulary set W using a stop word set S to obtain a second vocabulary set W', and further includes:

[0021] Load the predefined stop word set S, traverse the first vocabulary set W, and for each word w in the set, check whether the word belongs to the stop word set S;

[0022] If the word w is not in the stop word set S, then the word w is added to the second vocabulary set W'.

[0023] According to a specific implementation of the embodiment of the present invention, performing part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P includes:

[0024] Using a part-of-speech tagging tool provided for the second vocabulary set W′, performing a batch part-of-speech tagging operation, analyzing each vocabulary w through the part-of-speech tagging tool, and assigning one or more part-of-speech tags p to it;

[0025] For each word w, a bigram (w, p) is generated, and the part-of-speech tagging results of all words are summarized into a part-of-speech tagging result set P, expressed as P={(w, p)|w∈W′, p∈POS}.

[0026] According to a specific implementation of the embodiment of the present invention, the step of calculating the context of each word in the second vocabulary set W' to obtain the context set C includes:

[0027] For each word w in the second vocabulary set W′, take nc words before and after it as context words;

[0028] For each word w, determine the set of context words around it and record the order and position information of these words;

[0029] The context information of all words w is aggregated into a context set C, expressed as C={Ctx(T,k)}, where Ctx(T,k) represents the context information of word k in text T.

[0030] According to a specific implementation of the embodiment of the present invention, the second vocabulary set W' keyword k is mapped into a semantic vector V by the pre-trained model encoder. k , the semantic matrix M corresponding to the second vocabulary set W' is obtained, including:

[0031] Using the encoder part of the pre-trained model, each word in the second vocabulary set W′ is mapped to a fixed-length vector V k , vector V k It captures the position and characteristics of word k in the semantic space;

[0032] The semantic vector V of all words k k Summarize into a matrix M, each row of the matrix M corresponds to the semantic vector of a word, and the size of the matrix M is |W′|×d, where |W′| is the size of the vocabulary set W′ and d is the dimension of the semantic vector.

[0033] According to a specific implementation of an embodiment of the present invention, the generating of an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S comprises:

[0034] Calculating part-of-speech scores :

[0035]

[0036] are the weight coefficients of different parts of speech, is an indicator function, if vocabulary w is equal to vocabulary k, then The value of is 1, otherwise it is 0;

[0037] Calculating context score :

[0038]

[0039] d(c,k) is the distance between word k and its context c, is the attenuation coefficient;

[0040] Calculating semantic scores :

[0041]

[0042] is the semantic vector of word k Semantic vectors with other words The distance between is the adjustment coefficient;

[0043] Calculating similarity scores :

[0044]

[0045] s(k,w) is the similarity score between word k and other words w.

[0046] According to a specific implementation of an embodiment of the present invention, the method of generating an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S also includes:

[0047] Calculate the composite score :

[0048]

[0049] is the exponential coefficient, which is used to adjust the nonlinearity of the comprehensive score;

[0050] Select the score above the threshold The vocabulary of the extended word set T':

[0051] .

[0052] In a second aspect, an embodiment of the present invention provides a keyword expansion device based on a pre-training model, comprising:

[0053] The sorting module performs a word segmentation operation on the input text T to obtain a first vocabulary set W, and uses the stop word set S to perform a stop word removal operation on each word w in the vocabulary set W to obtain a second vocabulary set W'={w∈W|w S};

[0054] The computing module performs part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P={(w,p)|w∈W',p∈POS}, where POS represents a part-of-speech tag set, and calculates the context of each word in the second vocabulary set W' to obtain a context set C=Ctx(T,k), where the Ctx function is used to extract context information related to the keyword k;

[0055] Mapping module, which maps the second vocabulary set W'keyword k into a semantic vector V through the pre-trained model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W';

[0056] A similarity module calculates the similarity between each keyword k and other words in the second vocabulary set W' to obtain a similarity vector S;

[0057] A generation module generates an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S.

[0058] In a third aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0059] at least one processor; and,

[0060] a memory communicatively connected to the at least one processor; wherein,

[0061] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the keyword expansion method based on the pre-trained model in any of the aforementioned first aspects or any of the implementations of the first aspect.

[0062] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the keyword expansion method based on the pre-trained model in the aforementioned first aspect or any implementation of the first aspect.

[0063] In the fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the keyword expansion method based on the pre-trained model in the aforementioned first aspect or any implementation of the first aspect.

[0064] The keyword expansion scheme based on the pre-training model in the embodiment of the present invention includes: performing a word segmentation operation on the input text T to obtain a first vocabulary set W, using a stop word set S to perform a stop word removal operation on each word w in the vocabulary set W, and obtaining a second vocabulary set W'={w∈W|w S}; perform part-of-speech tagging on each word in the second vocabulary set W', and obtain the part-of-speech tagging result P={(w,p)|w∈W',p∈POS}, where POS represents the part-of-speech tag set; calculate the context of each word in the second vocabulary set W', and obtain the context set C=Ctx(T,k), where the Ctx function is used to extract context information related to keyword k; map the keyword k in the second vocabulary set W' to a semantic vector V through the pre-trained model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W'; calculate the similarity between each keyword k and other words in the second vocabulary set W' to obtain the similarity vector S; based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S, generate the extended word set T'=Fg(T|P,C,M,S) corresponding to the input text T. The present invention has the following beneficial effects:

[0065] aDeep semantic understanding:

[0066] The keyword expansion algorithm based on the custom model algorithm can capture the deep semantic relationship between words and generate more accurate and diverse expansion words.

[0067] bContext sensitivity:

[0068] The algorithm can generate keyword expansion results that are adapted to different contexts based on the specific context, enhancing the actual application effect of keywords, especially in scenarios that require contextual understanding such as social listening.

[0069] cReal-time and dynamic:

[0070] Through the dynamic update module, the system can obtain the latest data in a timely manner, generate extended words that conform to current language trends, keep the results cutting-edge, and adapt to the rapidly changing language environment.

[0071] dAutomation and efficiency:

[0072] From input processing to result generation and updating, the entire process is highly automated, which significantly improves the efficiency and accuracy of keyword expansion while reducing the need for manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 A flowchart of a keyword expansion method based on a pre-training model provided in an embodiment of the present invention;

[0075] Figure 2 A flowchart of another keyword expansion method based on a pre-training model provided in an embodiment of the present invention;

[0076] Figure 3 A schematic diagram of the structure of a keyword expansion device based on a pre-training model provided in an embodiment of the present invention;

[0077] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0079] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0080] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0081] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0082] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0083] The embodiment of the present disclosure provides a keyword expansion method based on a pre-trained model. The keyword expansion method based on a pre-trained model provided in this embodiment can be executed by a computing device, which can be implemented as software, or as a combination of software and hardware, and the computing device can be integrated in a server, terminal device, etc.

[0084] See also Figure 1 and Figure 2 The present disclosure provides a keyword expansion method based on a pre-training model, including:

[0085] S101, perform a word segmentation operation on the input text T to obtain a first vocabulary set W, use the stop word set S to perform a stop word removal operation on each word w in the vocabulary set W, and obtain a second vocabulary set W'={w∈W|w S}.

[0086] Before starting word segmentation, the input text T needs to be cleaned up to remove noise data, such as punctuation, special characters, numbers, etc. These data usually do not carry semantic information, but will interfere with the word segmentation effect. Punctuation in the text can be replaced with spaces or deleted directly. For example, "Hello, world!" is converted to "Helloworld". The text can be converted to a unified lowercase form to avoid the same word being mistaken for different words due to different case. For example, "Apple" and "apple" are unified as "apple". If the input text T contains multiple languages, the main language of the text can be identified first.

[0087] Use the selected word segmentation tool to segment the text T. The word segmentation tool will identify word boundaries based on language characteristics and dictionaries and divide the text into independent words.

[0088] For English text, the word segmentation tool will segment words based on spaces and punctuation. For example, "I lovenatural language processing." will be segmented into "I", "love", "natural", "language", "processing".

[0089] Chinese word segmentation: For Chinese text, the word segmentation tool will divide the words according to the dictionary and context. For example, "I love natural language processing." will be segmented into "I", "love", "natural language processing".

[0090] The results after word segmentation are summarized into a first vocabulary set W, which contains all independent vocabulary units. For example, for the English text "I love natural language processing", the vocabulary set W obtained after word segmentation is {"i", "love", "natural", "language", "processing"}.

[0091] Load the predefined stop word set S. This set usually contains a series of words that are considered to carry no significant information in text processing, such as "the", "is", "at", "which" in English.

[0092] An empty set W' can be created to store the vocabulary set after the stop words are removed. The vocabulary set W is traversed, and for each word w in the set, it is checked whether the word belongs to the stop word set S.

[0093] If the word w is not in the stop word set S (i.e., w∉S), then the word is considered to be a meaningful word and should be retained in the vocabulary set. The word w that is not a stop word is added to the second vocabulary set W'. Continue to perform the above judgment and addition operations on all words in the vocabulary set W until all words are traversed.

[0094] When all words are checked and processed, the second word set W' contains all non-stop words, expressed as W'={w∈W | w∉S}. Finally, the second word set W' after the stop words are removed is output as the basis for subsequent processing (such as part-of-speech tagging, context analysis, etc.).

[0095] Through the above steps, the input text T can be effectively segmented, and the stop word set S can be used to perform a stop word removal operation on each word w in the vocabulary set W, and finally a second vocabulary set W' is obtained, ensuring that the vocabulary in subsequent processing is more representative and reducing the impact of noise data.

[0096] S102, perform part-of-speech tagging on each word in the second vocabulary set W', and obtain the part-of-speech tagging result P={(w,p)|w∈W',p∈POS}, where POS represents the part-of-speech tag set, calculate the context of each word in the second vocabulary set W', and obtain the context set C=Ctx(T,k), where the Ctx function is used to extract context information related to the keyword k.

[0097] After the second vocabulary set W' is prepared, each word w in the second vocabulary set W' is tagged with a part-of-speech tag using a selected part-of-speech tagging tool. The part-of-speech tagging tool assigns one or more possible part-of-speech tags to each word w. Part-of-speech tags usually include nouns, verbs, adjectives, adverbs, etc.

[0098] The part-of-speech tagging results of all words w are aggregated into a part-of-speech tagging result set P, which is expressed as:

[0099] P={(w,p)|w∈W',p∈POS}, where POS represents the set of all possible part-of-speech tags.

[0100] Choose a suitable context window size nc, which determines how many neighboring words are considered around each word. For example, if the window size is 2, the first two and the last two words around each word will be considered. For each word w in the second vocabulary set W', take nc words before and after it as context words.

[0101] For each word w, determine the set of context words around it and record the order and position information of these words. The context words can be represented as an ordered list or sequence. Define a function Ctx (T, k) to extract context information related to keyword k. This function returns the set of context words based on the position of word k in text T.

[0102] Traverse each word w in the second word set W' and use the context function Ctx (T, w) to calculate the context word set of word w. Summarize the context information of all words w into a context set C, which is expressed as:

[0103]

[0104] Through the above steps, part-of-speech tagging can be effectively performed on each word in the second vocabulary set W', and the context of each word in the second vocabulary set W' can be calculated to obtain the part-of-speech tagging result P and the context set C.

[0105] S103, mapping the second vocabulary set W'keyword k into a semantic vector V through the pre-trained model encoder k , and obtain the semantic matrix M corresponding to the second vocabulary set W'.

[0106] You can set up a pre-trained model suitable for the target language. Load the selected pre-trained model. This usually involves loading the model's weight file and the corresponding configuration file.

[0107] Since pre-trained models usually have input length limits, the vocabulary set W' needs to be divided into multiple batches for processing. Each batch contains a certain number of words to ensure that the input length does not exceed the maximum limit of the model.

[0108] For each vocabulary w, construct an input sequence. Typically, the input sequence will contain special start and end tags (such as `[CLS]` and `[SEP]`) to help the model better understand the input.

[0109] For each word w in the second vocabulary set W', it is mapped to a fixed-length vector Vw using the encoder part of the pre-trained model. This involves encoding the word into the input format required by the model and getting the output of the hidden layer through the model.

[0110] The pre-trained model usually outputs a hidden state vector for each word, and the hidden state vector corresponding to each word can be extracted from the output of the model. For example, in BERT, the first hidden state vector after `[CLS]` is usually extracted as the semantic representation of the entire input sequence.

[0111] The semantic vectors Vw of all words w are aggregated into a matrix M, expressed as:

[0112]

[0113] Among them, v i It is the vocabulary w i The semantic vector of |W'| is the size of the vocabulary set W'.

[0114] Through the above steps, the pre-trained model encoder can be effectively used to map the keywords in the second vocabulary set W' into semantic vectors, and obtain the semantic matrix M corresponding to the second vocabulary set W'. This process helps to capture the position of the vocabulary in the semantic space and provide important semantic information for subsequent natural language processing tasks.

[0115] S104, calculating the similarity between each keyword k and other words in the second vocabulary set W' to obtain a similarity vector S.

[0116] For each keyword k in the second vocabulary set W', calculate the cosine similarity between it and other words. The specific steps are as follows:

[0117] 1. Extract vocabulary vector: Extract the semantic vector V of keyword k from the semantic matrix M k .

[0118] 2. Calculate similarity: For each word w (including k itself), calculate v k With v w The cosine similarity between:

[0119]

[0120] in, Represents vector v k and v w The dot product of |v k \ and |v w | respectively represent vector V k and v w The norm (length) of .

[0121] For each keyword k, the calculated similarity values ​​are summarized into a similarity vector s(k,·), expressed as:

[0122] s(k,⋅)={s(k,w)∣w∈W′}.

[0123] The similarity vectors s(k, ) of all keywords k are aggregated into a similarity vector set S, which is expressed as:

[0124] S = {s(k,⋅)|k∈W′}

[0125] Through the above steps, the similarity between each keyword k and other words in the second vocabulary set W' can be effectively calculated, and the similarity vector set S is obtained.

[0126] S105, based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S, generate an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T.

[0127] The comprehensive score can be To evaluate the importance of each word k, the formula takes into account factors such as part of speech, context, semantics, and similarity, including:

[0128] Calculating part-of-speech scores :

[0129]

[0130] are the weight coefficients of different parts of speech, is an indicator function, if vocabulary w is equal to vocabulary k, then The value of is 1, otherwise it is 0;

[0131] Calculating context score :

[0132]

[0133] d(c,k) is the distance between word k and its context c, is the attenuation coefficient;

[0134] Calculating semantic scores :

[0135]

[0136] is the semantic vector of word k Semantic vectors with other words The distance between is the adjustment coefficient;

[0137] Calculating similarity scores :

[0138]

[0139] s(k,w) is the similarity score between word k and other words w.

[0140] Calculate the composite score :

[0141]

[0142] is the exponential coefficient, which is used to adjust the nonlinearity of the comprehensive score;

[0143] Select the score above the threshold The vocabulary of the extended word set T':

[0144] .

[0145] Through the above detailed steps, the expanded word set T' corresponding to the input text T can be generated based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S.

[0146] See also Figure 2 The present invention also provides another keyword expansion method. Specifically, after the user inputs a keyword, the system first pre-processes the input to ensure that the generated expanded keyword is highly matched with the user's needs. The steps of the input processing module are as follows:

[0147] a participle:

[0148] Use word segmentation tools (such as NLTK, Spacy) to segment the input text into words and extract key components. For languages ​​without spaces such as Chinese, use statistical and rule-based word segmentation methods to segment sentences into words.

[0149] b Remove stop words:

[0150] Use the stop word list to remove common stop words (such as "is", "are", etc.) that have little impact on semantics, and retain the core keywords. This step helps reduce noise data and makes subsequent semantic analysis more accurate.

[0151] cPart of speech tagging:

[0152] Perform part-of-speech tagging on keywords (such as nouns, verbs, adjectives, etc.). This step uses the tagging tools provided by GPT-4 to help the model better understand the semantics and grammatical roles of keywords, thereby generating expanded words that are more in line with the context.

[0153] dContext analysis:

[0154] If the keyword comes from a text or sentence, the system will analyze the context in which it is located and extract contextual information. This process relies on GPT's context encoding ability to ensure that the expanded keywords are consistent with the overall semantics of the input content.

[0155] The keyword semantic expansion module is the core part of the present invention, which generates semantic expansion words related to the input keyword through GPT. The specific steps are as follows:

[0156] aSemantic vector generation:

[0157] GPT-4 is used to encode the input keywords into semantic vectors to capture their positions in the high-dimensional semantic space. This process is achieved through the encoder of GPT-4, and the keywords are represented as fixed-length vectors that reflect the meaning of the keywords and their relationship with other words in the semantic space.

[0158] bSemantic similarity calculation:

[0159] In the semantic space, the system calculates the cosine similarity or Euclidean distance between the input keyword and other words to generate a list of words with similar semantics to the input keyword. The specific calculation formula is as follows:

[0160]

[0161] Among them, 𝑣𝑖 and 𝑣𝑗 are the semantic vectors of the input keyword and potential expansion word respectively, ⋅ represents the dot product of the vector, and ∥𝑣𝑖∥ represents the norm of the vector. In this way, the system can filter out the expansion word that is closest to the input keyword in semantics.

[0162] c context generation:

[0163] For application scenarios that require the generation of keywords based on specific contexts (such as social listening or content generation), the present invention uses GPT's context generation capabilities to generate an extended keyword list suitable for different contexts. GPT-4 can generate a variety of relevant expressions based on contextual prompts, helping users find the most suitable keywords in different contexts.

[0164] d. Improved semantic diversity:

[0165] In order to increase the diversity of the expansion results, the system introduces strategies such as "Top-k Sampling" or "TemperatureScaling" to extract semantically similar but diverse words from the GPT generation model. This not only ensures the relevance of the expansion results, but also improves the breadth and diversity of keywords.

[0166] In order to ensure the quality of the expanded keywords, the present invention filters and sorts the generated keyword list. The specific steps are as follows:

[0167] a Filter irrelevant words:

[0168] The system first filters out words with large semantic differences from the input keyword based on the semantic similarity threshold or manually defined rules, and retains highly relevant expansion words. For example, only words with a similarity to the input keyword exceeding a certain predetermined value (such as 0.7) will be retained.

[0169] b. Frequency analysis:

[0170] By analyzing the frequency of keyword usage in external data sources (such as social media data or search engine logs), the system selects more commonly used and popular words. This step is achieved by connecting to external databases (such as Google Trends API) to ensure that the expansion results are in line with current usage trends and user habits.

[0171] c Contextual adaptability:

[0172] For specific industries or scenarios (such as legal, medical and other professional fields), the system further selects keywords that match the specific context of the field by comparing the words used in the professional corpus. This process uses the field corpus and the expanded words for matching analysis to ensure the professionalism and adaptability of the results.

[0173] d sort:

[0174] The final keyword list is sorted based on multi-dimensional indicators (such as semantic similarity, keyword usage frequency, and contextual matching). The system uses a weighted scoring model to comprehensively calculate these indicators and ultimately displays the expansion words that are most relevant to the input keyword and meet user needs. The sorting formula is as follows:

[0175]

[0176] Among them, 𝑤1, 𝑤2, and 𝑤3 are the weights of different indicators, which are dynamically adjusted according to user needs and application scenarios to ensure that the most suitable keywords are ranked at the top.

[0177] In order to maintain the real-time and cutting-edge nature of keyword expansion, the present invention introduces a dynamic update mechanism. The specific steps are as follows:

[0178] a Real-time data collection:

[0179] The system regularly obtains the latest text data from external data sources (such as news websites, social media platforms, and industry report databases) through API interfaces and incorporates this data into the update process of the keyword library. This data can help identify new buzzwords and industry terms to keep the keyword library cutting-edge.

[0180] b Keyword library update:

[0181] The system regularly updates its internal keyword library using the latest data from external data sources. The update process includes adding new words, deleting or adjusting old words, and fine-tuning the GPT expansion model to enable it to recognize and generate new keywords.

[0182] c Model fine-tuning:

[0183] Based on the new data, the GPT model is continuously fine-tuned to enable it to adapt to changes in language and vocabulary and generate keyword expansion results that are more in line with current trends. The fine-tuning process includes retraining the model with new data and adjusting the model's parameters based on the training results.

[0184] d Feedback mechanism:

[0185] The present invention also introduces a user feedback mechanism to adjust the parameters and filtering criteria of the keyword expansion algorithm according to the actual user feedback, so as to improve the practicality and user satisfaction of the expansion results. User feedback can be collected through user interaction data in the interface or application, and the system will dynamically optimize the model and strategy based on the feedback.

[0186] The present invention has the following advantages:

[0187] a Deep semantic understanding:

[0188] The GPT-based keyword expansion algorithm can capture the deep semantic relationship between words and generate more accurate and diverse expansion words.

[0189] bContext sensitivity:

[0190] The algorithm can generate keyword expansion results that are adapted to different contexts based on the specific context, enhancing the actual application effect of keywords, especially in scenarios that require contextual understanding such as social listening.

[0191] cReal-time and dynamic:

[0192] Through the dynamic update module, the system can obtain the latest data in a timely manner, generate extended words that conform to current language trends, keep the results cutting-edge, and adapt to the rapidly changing language environment.

[0193] dAutomation and efficiency:

[0194] From input processing to result generation and updating, the entire process is highly automated, which significantly improves the efficiency and accuracy of keyword expansion while reducing the need for manual intervention.

[0195] According to a specific implementation of an embodiment of the present invention, the word segmentation operation is performed on the input text T to obtain a first vocabulary set W, and a stop word set S is used to perform a stop word removal operation on each word w in the vocabulary set W to obtain a second vocabulary set W', including:

[0196] After removing special characters from the input text T, the language type of the input text T is detected;

[0197] Obtaining n word segmentation dictionaries corresponding to the language type, and generating n different word segmentation results based on the n word segmentation dictionaries;

[0198] The n different word segmentation results are removed and merged to obtain the first vocabulary set W.

[0199] According to a specific implementation of an embodiment of the present invention, the word segmentation operation is performed on the input text T to obtain a first vocabulary set W, and a stop word removal operation is performed on each word w in the vocabulary set W using a stop word set S to obtain a second vocabulary set W', and further includes:

[0200] Load the predefined stop word set S, traverse the first vocabulary set W, and for each word w in the set, check whether the word belongs to the stop word set S;

[0201] If the word w is not in the stop word set S, then the word w is added to the second vocabulary set W'.

[0202] According to a specific implementation of the embodiment of the present invention, performing part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P includes:

[0203] Using a part-of-speech tagging tool provided for the second vocabulary set W′, performing a batch part-of-speech tagging operation, analyzing each vocabulary w through the part-of-speech tagging tool, and assigning one or more part-of-speech tags p to it;

[0204] For each word w, a bigram (w, p) is generated, and the part-of-speech tagging results of all words are summarized into a part-of-speech tagging result set P, expressed as P={(w, p)|w∈W′, p∈POS}.

[0205] According to a specific implementation of the embodiment of the present invention, the step of calculating the context of each word in the second vocabulary set W' to obtain the context set C includes:

[0206] For each word w in the second vocabulary set W′, take nc words before and after it as context words;

[0207] For each word w, determine the set of context words around it and record the order and position information of these words;

[0208] The context information of all words w is aggregated into a context set C, expressed as C={Ctx(T,k)}, where Ctx(T,k) represents the context information of word k in text T.

[0209] According to a specific implementation of the embodiment of the present invention, the second vocabulary set W' keyword k is mapped into a semantic vector V by the pre-trained model encoder. k , the semantic matrix M corresponding to the second vocabulary set W' is obtained, including:

[0210] Using the encoder part of the pre-trained model, each word in the second vocabulary set W′ is mapped to a fixed-length vector V k , vector V k It captures the position and characteristics of word k in the semantic space;

[0211] The semantic vector V of all words k k Summarize into a matrix M, each row of the matrix M corresponds to the semantic vector of a word, and the size of the matrix M is |W′|×d, where |W′| is the size of the vocabulary set W′ and d is the dimension of the semantic vector.

[0212] According to a specific implementation of an embodiment of the present invention, the generating of an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S comprises:

[0213] Calculating part-of-speech scores :

[0214]

[0215] are the weight coefficients of different parts of speech, is an indicator function, if vocabulary w is equal to vocabulary k, then The value of is 1, otherwise it is 0;

[0216] Calculating context score :

[0217]

[0218] d(c,k) is the distance between word k and its context c, is the attenuation coefficient;

[0219] Calculating semantic scores :

[0220]

[0221] is the semantic vector of word k Semantic vectors with other words The distance between is the adjustment coefficient;

[0222] Calculating similarity scores :

[0223]

[0224] s(k,w) is the similarity score between word k and other words w.

[0225] According to a specific implementation of an embodiment of the present invention, the method of generating an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S also includes:

[0226] Calculate the composite score :

[0227]

[0228] is the exponential coefficient, which is used to adjust the nonlinearity of the comprehensive score;

[0229] Select the score above the threshold The vocabulary of the extended word set T':

[0230] .

[0231] Corresponding to the above method embodiment, see Figure 3 The embodiment of the present invention further discloses a keyword expansion device 30 based on a pre-training model, comprising:

[0232] Arrangement module 301 performs a word segmentation operation on input text T to obtain a first vocabulary set W, and uses a stop word set S to perform a stop word removal operation on each word w in the vocabulary set W to obtain a second vocabulary set W'={w∈W|w S};

[0233] The calculation module 302 performs part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P={(w,p)|w∈W',p∈POS}, where POS represents a part-of-speech tag set, and calculates the context of each word in the second vocabulary set W' to obtain a context set C=Ctx(T,k), where the Ctx function is used to extract context information related to the keyword k;

[0234] Mapping module 303 maps the second vocabulary set W' keyword k into a semantic vector V through a pre-trained model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W';

[0235] Similarity module 304, calculating the similarity between each keyword k and other words in the second vocabulary set W' to obtain a similarity vector S;

[0236] The generation module 305 generates an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S.

[0237] See also Figure 4 The embodiment of the present invention further provides an electronic device 60, the electronic device comprising:

[0238] at least one processor; and,

[0239] a memory communicatively connected to the at least one processor; wherein,

[0240] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the keyword expansion method based on the pre-training model in the aforementioned method embodiment.

[0241] An embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the aforementioned method embodiment.

[0242] An embodiment of the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the keyword expansion method based on the pre-trained model in the aforementioned method embodiment.

[0243] Reference below Figure 4 , which shows a schematic diagram of the structure of an electronic device 60 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0244] like Figure 4As shown, the electronic device 60 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 60 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0245] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, a gage, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 60 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 60 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

[0246] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0247] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A keyword expansion method based on a pre-training model, characterized in that: include: Perform word segmentation on the input text T to obtain the first vocabulary set W, and use the stop word set S to perform stop word removal operation on each word w in the vocabulary set W to obtain the second vocabulary set W'={w∈W|w S}; Perform part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P = {(w, p) | w∈W', p∈POS}, where POS represents a set of part-of-speech tags. Calculate the context of each word in the second vocabulary set W' to obtain a context set C = Ctx(T, k). The Ctx function is used to extract context information related to keyword k; The second vocabulary set W' keyword k is mapped to a semantic vector V through the pre-trained model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W'; Calculate the similarity between each keyword k and other words in the second vocabulary set W' to obtain a similarity vector S; Based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S, an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T is generated, including: Calculating part-of-speech scores : are the weight coefficients of different parts of speech, is an indicator function, if vocabulary w is equal to vocabulary k, then The value of is 1, otherwise it is 0; Calculating context score : d(c,k) is the distance between word k and its context c, is the attenuation coefficient; Calculating semantic scores : is the semantic vector of word k Semantic vectors with other words The distance between is the adjustment coefficient; Calculating similarity scores : s(k,w) is the similarity score between word k and other words w; Calculate the composite score : is the exponential coefficient, which is used to adjust the nonlinearity of the comprehensive score; Select the score above the threshold The vocabulary of the extended word set T': 。 2. The method according to claim 1, characterized in that The word segmentation operation is performed on the input text T to obtain a first vocabulary set W, and a stop word removal operation is performed on each word w in the vocabulary set W using the stop word set S to obtain a second vocabulary set W', including: After removing special characters from the input text T, the language type of the input text T is detected; Obtaining n word segmentation dictionaries corresponding to the language type, and generating n different word segmentation results based on the n word segmentation dictionaries; The n different word segmentation results are removed and merged to obtain the first vocabulary set W.

3. The method according to claim 2, characterized in that The method of performing a word segmentation operation on the input text T to obtain a first vocabulary set W, performing a stop word removal operation on each word w in the vocabulary set W using the stop word set S to obtain a second vocabulary set W', further comprising: Load the predefined stop word set S, traverse the first vocabulary set W, and for each word w in the set, check whether the word belongs to the stop word set S; If the word w is not in the stop word set S, then the word w is added to the second vocabulary set W'.

4. The method according to claim 3, characterized in that The performing part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P includes: Using a part-of-speech tagging tool provided for the second vocabulary set W′, performing a batch part-of-speech tagging operation, analyzing each vocabulary w through the part-of-speech tagging tool, and assigning one or more part-of-speech tags p to it; For each word w, a bigram (w, p) is generated, and the part-of-speech tagging results of all words are summarized into a part-of-speech tagging result set P, expressed as P={(w, p)|w∈W′, p∈POS}.

5. The method according to claim 4, characterized in that The step of calculating the context of each word in the second vocabulary set W' to obtain a context set C includes: For each word w in the second vocabulary set W′, take nc words before and after it as context words; For each word w, determine the set of context words around it and record the order and position information of these words; The context information of all words w is aggregated into a context set C, expressed as C={Ctx(T,k)}, where Ctx(T,k) represents the context information of word k in text T.

6. The method according to claim 5, characterized in that The second vocabulary set W' keyword k is mapped into a semantic vector V by the pre-trained model encoder k , the semantic matrix M corresponding to the second vocabulary set W' is obtained, including: Using the encoder part of the pre-trained model, each word in the second vocabulary set W′ is mapped to a fixed-length vector V k , vector V k It captures the position and characteristics of word k in the semantic space; The semantic vector V of all words k k Summarize into a matrix M, each row of the matrix M corresponds to the semantic vector of a word, and the size of the matrix M is |W′|×d, where |W′| is the size of the vocabulary set W′ and d is the dimension of the semantic vector.

7. A keyword expansion device based on a pre-training model, characterized in that: include: The sorting module performs a word segmentation operation on the input text T to obtain a first vocabulary set W, and uses the stop word set S to perform a stop word removal operation on each word w in the vocabulary set W to obtain a second vocabulary set W'={w∈W|w S}; The computing module performs part-of-speech tagging on each word in the second vocabulary set W' to obtain a part-of-speech tagging result P={(w,p)|w∈W',p∈POS}, where POS represents a part-of-speech tag set, and calculates the context of each word in the second vocabulary set W' to obtain a context set C=Ctx(T,k), where the Ctx function is used to extract context information related to the keyword k; Mapping module, which maps the second vocabulary set W'keyword k into a semantic vector V through the pre-trained model encoder k , obtain the semantic matrix M corresponding to the second vocabulary set W'; A similarity module calculates the similarity between each keyword k and other words in the second vocabulary set W' to obtain a similarity vector S; A generation module generates an expanded word set T'=Fg(T|P,C,M,S) corresponding to the input text T based on the part-of-speech tagging result P, the context set C, the semantic matrix M and the similarity vector S, including: Calculating part-of-speech scores : are the weight coefficients of different parts of speech, is an indicator function, if vocabulary w is equal to vocabulary k, then The value of is 1, otherwise it is 0; Calculating context score : d(c,k) is the distance between word k and its context c, is the attenuation coefficient; Calculating semantic scores : is the semantic vector of word k Semantic vectors with other words The distance between is the adjustment coefficient; Calculating similarity scores : s(k,w) is the similarity score between word k and other words w; Calculate the composite score : is the exponential coefficient, which is used to adjust the nonlinearity of the comprehensive score; Select the score above the threshold The vocabulary of the extended word set T': 。 8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the keyword expansion method based on the pre-training model as described in any one of claims 1 to 6.

Images