Synonym acquisition method, search method, device, storage medium and program product

Abstract

The embodiment of the invention provides a synonym obtaining method, a searching method, equipment, a storage medium and a program product, and relates to the technical field of computers, the method comprises the steps that core words are extracted from received corpus data, and the core words are used for indicating text features of the corpus data; performing vectorization processing on the extracted core word by using a plurality of pre-trained different word vector models to obtain a plurality of word vectors of the core word, the different word vector models being used for performing vectorization processing on different features of the core word; and generating synonyms of the core word based on the plurality of word vectors. According to the technical scheme, the synonyms in the corpus data can be automatically obtained, the accuracy of the obtained synonyms is improved, meanwhile, the manual searching and screening time can be shortened, and therefore the labor cost can be reduced, and the processing efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a synonym acquisition method, search method, device, storage medium, and program product. Background Technology

[0002] In search scenarios, synonyms play a crucial role in understanding user intent. Because users have diverse search habits, the keywords they use may vary. With the support of synonyms, even if the user's search terms do not perfectly match the terms recorded in the search engine, the system can still present search results related to those terms, greatly improving search efficiency and accuracy.

[0003] Synonym mining is a common method in the field of natural language processing, and a high-quality thesaurus can provide better services to users. However, to improve the accuracy of synonym mining results, manual review is often required, which is costly and time-consuming. Summary of the Invention

[0004] This application addresses the problem of high cost and time consumption of manual synonym mining in the aforementioned related technologies by proposing a synonym acquisition method, search method, device, storage medium, and program product.

[0005] The first aspect of this application proposes a method for obtaining synonyms, the method comprising: extracting core words from received corpus data, the core words being used to indicate the textual features of the corpus data; vectorizing the extracted core words using multiple pre-trained word vector models to obtain multiple word vectors of the core words, the different word vector models being used to vectorize different features of the core words; and generating synonyms of the core words based on the multiple word vectors.

[0006] A second aspect of this application discloses a search method, comprising: receiving a search request, the search request including search content; performing word segmentation on the search content to obtain search terms in the search content; searching for synonyms of the search terms from a thesaurus, the thesaurus including: core words extracted from corpus data and synonyms of the core words, wherein the synonyms of the core words are words obtained by performing the method of the first aspect on the corpus data; and retrieving the search terms and the synonyms of the search terms.

[0007] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method described in the first or second aspect above.

[0008] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method as described in the first or second aspect above.

[0009] An embodiment of the fifth aspect of this application provides a computer program product, including a computer program that is executed by a processor to implement the methods described in the first or second aspect above.

[0010] Based on the synonym acquisition method described in the first aspect above, this application has at least the following beneficial effects or advantages:

[0011] The synonym acquisition method in this application utilizes multiple different word vector models to extract multiple word vectors for a core word. Different word vector models are used to vectorize different features of the core word. Therefore, multiple word vectors of the core word can represent multiple semantic features of the core word. Since multiple semantic features help cover a wider semantic range and improve the accuracy of semantic understanding, the synonyms generated subsequently based on multiple word vectors have a higher semantic similarity to the core word, which helps improve the accuracy of the acquired synonyms and thus obtain higher-quality synonyms. Furthermore, compared to related technologies where manual synonym mining is easily affected by subjective judgment, the method in this application can automatically acquire synonyms from corpus data. While improving the accuracy of the acquired synonyms, it can reduce the time spent on manual searching and filtering, thereby reducing labor costs and improving processing efficiency.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0014] Figure 1 A flowchart illustrating a synonym acquisition method according to an embodiment of this application;

[0015] Figure 2 A flowchart illustrating a synonym acquisition method according to an exemplary embodiment of this application;

[0016] Figure 3 A flowchart illustrating a search method according to an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of the device for obtaining synonyms according to this application;

[0018] Figure 5 This is a schematic diagram of the search device according to this application;

[0019] Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an exemplary embodiment of this application;

[0020] Figure 7 This is a schematic diagram illustrating the structure of a storage medium according to an exemplary embodiment of this application. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination," etc.

[0024] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0025] In related technologies, in order to improve the accuracy of synonym mining results, missing synonyms often need to be manually reviewed. This is time-consuming and labor-intensive, and is easily affected by human subjective judgment, thus affecting the accuracy of the mining results.

[0026] Based on this, embodiments of this application provide a synonym acquisition method to improve the accuracy of the acquired synonyms, and to help reduce labor costs and improve processing efficiency.

[0027] The synonym acquisition method according to the embodiments of this application can be executed by electronic devices such as terminal devices or servers. Terminal devices may include, but are not limited to, user equipment (UE), mobile devices, tablet computers, personal digital assistants (PDAs), laptops, and desktop computers. This method can be implemented by a processor calling computer-readable program instructions stored in memory. Servers may include independent physical servers, server clusters consisting of multiple servers, or cloud servers capable of cloud computing.

[0028] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0029] Figure 1 A flowchart of the synonym acquisition method provided in the embodiments of this application is shown below. Figure 1 As shown, the method specifically includes the following steps S110-S130.

[0030] S110, extract core words from the received corpus data. Core words are used to indicate the textual features of the corpus data.

[0031] S120 uses multiple pre-trained word vector models to vectorize the extracted core words, resulting in multiple word vectors for the core words. The word vectors are used to represent the semantic features of the core words, and different word vector models are used to vectorize different features of the core words.

[0032] S130 generates synonyms for core words based on multiple word vectors.

[0033] According to the synonym acquisition method of this application, multiple word vectors of a core word can be extracted using multiple different word vector models. Different word vector models are used to vectorize different features of the core word. Therefore, multiple word vectors of the core word can represent multiple semantic features of the core word. Since multiple semantic features help cover a wider semantic range and improve the accuracy of semantic understanding, the synonyms of the core word subsequently generated based on multiple word vectors have a higher semantic similarity to the core word, which helps improve the accuracy of the acquired synonyms, thus obtaining higher quality synonyms. Furthermore, compared to related technologies where manual synonym mining is easily affected by subjective judgment, the method of this application can automatically acquire synonyms from corpus data. While improving the accuracy of the acquired synonyms, it can reduce the time spent on manual searching and filtering, thereby reducing labor costs and improving processing efficiency.

[0034] In step S110, the sources of the corpus data can be various. For example, the corpus data may include at least one of the following: arbitrary corpus data sent by the customer and corpus data downloaded from the Internet. As an example, corpus data downloaded from the Internet may include at least one of the following: open Chinese corpora, encyclopedia entries, knowledge bases, online forums, and news websites, etc.

[0035] In step S110, text features are useful information extracted from the corpus data. For example, text features include at least one of the following: named entity type, part-of-speech tag, term frequency (TF), inverse document frequency (IDF), etc.

[0036] Named entity types are used to identify entities with specific meanings from corpus data. Named entity types may include at least one of the following: person names, place names, time, date, organization names, product names, and proper nouns. Parts of speech refer to the grammatical role of words in a sentence. Parts of speech may include at least one of the following: noun, verb, adjective, adverb, pronoun, etc.

[0037] In corpus data, the term frequency (TF) refers to the number of times the word appears in the corpus. In some scenarios, TF can be standardized for easier comparison. Specifically, the standardized TF value is obtained by dividing the total number of times a word appears in the corpus by the total number of words in the corpus. For example, if a word appears 2 times, 3 times, and 1 time in three documents in the corpus, then the word appears a total of 6 times. By segmenting the corpus into words, we can obtain the total number of words in the corpus. Assuming the total number of words in the corpus is 100, the standardized TF value for that word is 6 / 100 = 0.06. In practical scenarios, considering the varying document lengths in different corpora, longer documents result in a larger total word count; therefore, document length affects the TF statistics. Based on this, in the embodiments of this application, the standardized value of word frequency helps to eliminate the influence of document length in the corpus data, ensuring that word frequency is independent of the document length in the corpus data, making the comparison of word frequency of documents of different lengths more fair and accurate.

[0038] In this context, the inverse document frequency of a word in the corpus data refers to the logarithm of a predetermined ratio, which is the ratio of the total number of documents in the corpus data to the number of documents containing that word.

[0039] In the embodiments of this application, core words can be used to indicate the textual features of corpus data. They are words that play an important role in the search and are the carriers of key information in the corpus data. Extracting core words helps to identify valuable information in the text and facilitates the acquisition of high-quality synonyms.

[0040] In some embodiments, the corpus data does not include behavior logs, which refer to search records and user interactions with the search system or service, such as clicks, browsing, and purchases.

[0041] In related technologies, synonym mining methods usually rely on behavior logs, but behavior logs are often difficult to obtain. In this application embodiment, synonyms can be obtained based on corpus data without behavior logs, which can reduce the difficulty of data collection, protect user information security, reduce dependence on specific data, and improve the flexibility of synonym acquisition.

[0042] In some embodiments, the step of extracting core words from the received corpus data in step S110 may specifically include: performing text cleaning on the corpus data to obtain cleaned corpus data; performing word segmentation on the cleaned corpus data to obtain multiple words in the corpus data; and extracting core words from the multiple words.

[0043] In this embodiment, text cleaning is also referred to as data cleaning or data preprocessing. Text cleaning includes, but is not limited to, at least one of the following: removing stop words, removing duplicate content, and removing noisy data.

[0044] Specifically, removing stop words refers to removing words that frequently appear but do not usually represent specific things or concepts, such as function words (conjunctions, prepositions, pronouns, etc.), emoticons, and modal particles. Removing duplicate content refers to removing duplicate lines and words from the corpus data. Removing noisy data refers to removing irrelevant characters from the corpus data, such as HyperText Markup Language (HTML) tags and special characters. HTML tags include, but are not limited to, paragraph tags, link tags, and image tags. Special characters include, but are not limited to, punctuation marks, currency symbols, and mathematical symbols.

[0045] In this embodiment, word segmentation refers to dividing a continuous text string in the corpus data into individual words or phrases. Through word segmentation, multiple words in the corpus data can be obtained, and core words can be extracted from these multiple words.

[0046] In this embodiment, text cleaning of the corpus data can remove irrelevant or redundant information, improving data processing efficiency. By segmenting the corpus data into multiple words, long texts (e.g., sentences in the corpus data) can be divided into smaller processing units (words within sentences), thereby reducing the complexity of text processing, lowering the difficulty of subsequent core word extraction, and improving processing efficiency.

[0047] In some embodiments, the step of extracting core words from multiple words may specifically include: calculating the word frequency and inverse document frequency of each word in the corpus data; determining the weight value of the corresponding word based on the word frequency and inverse document frequency; obtaining words from the multiple words whose weight value is greater than or equal to a predetermined weight threshold to obtain the first core word; obtaining words from the multiple words that are included in a specific domain corpus to obtain the second core word; obtaining the part-of-speech and entity type of each word; obtaining words from the multiple words whose part-of-speech is a predetermined part-of-speech and whose entity type is a predetermined named entity type to obtain the third core word; and using the first core word, the second core word, and the third core word as core words extracted from the corpus data.

[0048] For example, regarding the word frequency and inverse document frequency (IVF) of a word in the corpus data, please refer to the description of word frequency and IVF in the above embodiments. In this embodiment, the product of word frequency and IVF is used as the weight value of the word, which can be used to evaluate the importance of the word in the corpus data. Words with higher weight values ​​are considered to be core words in the corpus data.

[0049] For example, a domain-specific corpus is a collection of text data pre-collected and organized for a specific discipline or industry. Such corpora typically contain terms and concepts related to that specific domain. Examples include, but are not limited to, corpora in the medical field, legal field, and scientific and technological field. If a word is found in a domain-specific corpus, it ensures that the word possesses a certain level of specialization and can be considered a core word in the corpus data.

[0050] For example, as can be seen from the description of parts of speech and named entity types in the above embodiments, both parts of speech and named entity types can include multiple types. As an example, a predetermined part of speech can be set to at least one of nouns and adjectives, and a predetermined named entity type can be set to at least one of person names, place names, and organization names.

[0051] In this embodiment, the first core word can represent words with high importance in the corpus data, the second core word can represent terms and concepts in a specific field, and the third core word can represent entities with clear referents in the corpus data. After removing duplicates from all three, they are all used as core words extracted from the corpus data, which is conducive to extracting core words from different perspectives and providing a diversified and high-quality data foundation for the subsequent acquisition of synonyms.

[0052] In step S120 above, "multiple different word vector models" refers to multiple word vector models of different types. For example, multiple different word vector models include, but are not limited to, at least two of the following: Bag of Words (BoW) model, word2vec model, Global Vectors for Word Representation (GloVe) model, Term Frequency-Inverse Document Frequency model, and models based on large-scale pre-trained models, etc.

[0053] In this step, word vector models are text representation models that transform text data into computer-readable vector forms for subsequent analysis and processing. By vectorizing text, the powerful computing capabilities of computers can be leveraged for in-depth analysis of text data.

[0054] In this step, different word vector models can be used to vectorize different features of the core word. After multiple different word vector models vectorize the core word, multiple word vectors of the core word are obtained. These multiple word vectors can be used to represent multiple different features of the core word.

[0055] For example, the bag-of-words model uses word frequency as a feature when converting words into corresponding word vectors, without considering word order, making it relatively simple, intuitive, and easy to implement. Word embedding models can convert words into word vectors, which are typically vectors composed of real numbers. Word vectors generated by word embedding models can capture rich linguistic features of words, including but not limited to semantic, grammatical, and syntactic information. Global information-based word vector representation models mainly utilize the global features of words in the corpus, i.e., the co-occurrence probability of words throughout the corpus, to generate word vectors. The co-occurrence probability refers to the probability that a word and other words in the corpus appear simultaneously at any position in the corpus. Generally, the higher the probability of simultaneous occurrence, the greater the semantic correlation between the two words. Therefore, the co-occurrence probability can be used to assess the degree of semantic association between words. Word vectors generated by the term frequency-inverse document frequency (IF-IVF) model combine both term frequency and IVF features. Word vectors generated through the IF-IVF model are helpful in assessing the importance of a word to a specific document in a corpus.

[0056] Among these, models based on large-scale pre-trained models include at least one of the following: Bidirectional Encoder Representations from Transformers (BERT) models, Generative Pre-trained Transformer (GPT) models, and Robustly Optimized BERT Pretraining Approach (RoBERTa) models. Models based on large-scale pre-trained models can capture the rich linguistic features and contextual information contained in words.

[0057] In this embodiment, by vectorizing any core word using different types of word vector models, multiple word vectors for that core word can be obtained. Different types of word vector models can capture different semantic features of a core word. Multiple word vectors are beneficial for a comprehensive understanding of the meaning of the core word. Furthermore, multiple word vectors are helpful for evaluating the similarity between words from different perspectives, which helps improve the accuracy of identifying synonyms of the core word.

[0058] In some embodiments, step S130 may specifically include: generating multiple candidate words that are semantically related to the core word based on multiple word vectors; and obtaining synonyms of the core word from the multiple candidate words.

[0059] In this embodiment, each candidate word is a word that has a semantic relationship with the core word. Multiple candidate words can enrich the selection range of synonyms for the core word. In some scenarios, if the core word is a polysemous word, then when multiple candidate words for the core word are generated, synonyms can be selected from the broader semantic range provided by the multiple candidate words, which is beneficial for more accurately obtaining synonyms for the core word.

[0060] In some embodiments, the step of generating multiple candidate words semantically related to a core word based on multiple word vectors may specifically include: performing semantic similarity-based vector retrieval in a predetermined vector set using multiple word vectors to obtain a first number of first candidate words, wherein the predetermined vector set includes at least multiple word vectors of each core word; inputting the core word into a predetermined generative model to obtain a second number of second candidate words, wherein the second candidate words are words outside the corpus data and the semantic similarity between the second candidate words and the core word is greater than a predetermined similarity threshold; and generating multiple candidate words semantically related to the core word based on at least one of the first number of first candidate words and the second number of second candidate words.

[0061] For example, vector recall can be understood as an information retrieval technology that can convert words into vectors, thereby measuring the degree of semantic similarity between words in the vector space by the similarity between vectors, and thus quickly retrieving words that are semantically related to the given word.

[0062] As a specific example, the process of vector recall is described as follows: First, multiple word vector models of different types can be used to vectorize the core word, resulting in multiple word vectors for the core word, with each word vector model corresponding to one word vector for the core word. Second, for any word vector among the multiple word vectors of the core word, the similarity between the word vector of the core word and each word vector in a predetermined vector set can be calculated. The similarity between word vectors can be used to determine the similarity between words. Next, using the calculated similarity between word vectors, at least one word that is semantically similar to the core word can be obtained as a word recalled by the word vector of the core word. Then, by merging the words recalled by the various word vectors of the core word, all words recalled by the multiple word vectors of the core word are obtained. After that, after deduplicating all the recalled words, they are sorted in descending order according to their semantic similarity to the core word, and the first number of words are selected from the sorted words as the first candidate words of the core word.

[0063] In the exemplary vector retrieval process described above, the predetermined vector set can be multiple word vectors for each core word, or it can be the word vectors for all words in the corpus data. Since the word vectors for all words in the corpus data contain multiple word vectors for each core word, it can be assumed that the predetermined vector set includes at least multiple word vectors for each core word.

[0064] In this embodiment, when the predetermined vector set consists of multiple word vectors of each core word, the multiple word vectors of the core words can be used to recall based on different semantic features of the core words, which is beneficial to improve the relevance of the recall results and speed up the recall process. When the predetermined vector set consists of word vectors of all words in the corpus data, the recall process involves word vectors of more words, which is beneficial to more comprehensively cover potential recall items and improve the recall rate.

[0065] For example, the generative model in the embodiments of this application can be an existing model for generating synonyms, or it can be a model trained on a large and diverse corpus. Through pre-training, the model can learn the rich linguistic features in the diverse corpus, thereby having the ability to understand complex language structures and generate new words.

[0066] As specific examples, diverse corpora include, but are not limited to, at least two of the following: encyclopedia content, news articles, books, web pages, social forums, blog posts, professional literature, etc.

[0067] In this embodiment, the candidate words for the core words generated by the generative model are new words that have not appeared in the corpus data. Therefore, the generative model can provide new words that vector recall methods may not cover, expand the selection range of candidate words for the core words, increase the diversity of the semantic space, and facilitate the finding of more diverse synonyms.

[0068] In this embodiment of the application, the vector recall method can also be combined with the generative model-based method. On the one hand, the vector recall method can retrieve the word vector that is most similar to the word vector of the core word from a predetermined vector set. On the other hand, the generative model can generate new candidate words. The combination of the two can improve the recall rate and increase the diversity of candidate word acquisition results.

[0069] In some embodiments, obtaining synonyms of the core word from a plurality of candidate words includes: for any candidate word among the plurality of candidate words, classifying the core word and the candidate word using a predetermined language model to obtain a classification result; and confirming the candidate word as a synonym of the core word if the classification result indicates that the core word and the candidate word are synonyms.

[0070] For example, a language model (LM) is a tool that can capture the deep semantic features of words, calculate the probability that a core word and a candidate word are synonyms, and determine that the two are synonyms if the probability value is greater than a predetermined probability threshold.

[0071] For example, a language model can be a large language model (LLM), which is a language model with a large number of parameters. Large language models are typically trained on large-scale datasets and are able to capture more granular language features and complex language patterns. Leveraging the powerful semantic understanding capabilities of large language models, binary classification is performed on the mined candidate words, that is, determining whether a pair of candidate words is a synonym, thereby obtaining synonyms with higher accuracy.

[0072] It should be understood that language models can have many different types. For example, a language model can also be a cosine similarity model, which can assess whether two word vectors are synonyms by calculating the cosine similarity between them; another example is a knowledge-based model, which can identify whether the relationship between two words is synonymous based on a knowledge base. The specific type of language model can be selected according to actual needs, and this application does not impose specific limitations on the embodiments.

[0073] In this embodiment, the powerful semantic understanding capability of the language model is used to determine whether the core word and candidate word are synonyms, which can save the tedious and laborious manual verification process and help improve the quality and accuracy of synonym pairs.

[0074] In some embodiments, before obtaining synonyms of the core word from multiple candidate words, the method further includes: filtering multiple candidate words of the core word based on at least one of named entity type and characters contained in the word; and / or filtering multiple candidate words of the core word based on text similarity.

[0075] In this embodiment, multiple candidate words are first filtered, and then synonyms of the core word are obtained from the filtered candidate words. Specifically, filtering based on named entity type and the characters contained in the words relies on explicit named instance types and the characters contained in the words themselves, making it easy to implement and allowing for fast execution of the filtering process, thus improving processing efficiency. Filtering based on text similarity can provide a deeper understanding of word meaning and offer more accurate synonym recognition. Both methods have their advantages and can be used in combination. For example, multiple candidate words can be filtered first based on named entity type and the characters contained in the words to quickly filter out obviously irrelevant candidate words. Then, a text similarity-based method can be used to perform a more in-depth analysis and screening of the remaining candidate words, thereby combining the speed and accuracy of the filtering process to improve the overall synonym recognition effect.

[0076] In some embodiments, the step of filtering multiple candidate words of a core word based on at least one of named entity type and characters contained in the word in the above embodiments may specifically include: for any candidate word among multiple candidate words of the core word, performing at least one of the following filtering processes: filtering out the candidate word when the named entity type of the core word and the named entity type of the candidate word are different types; filtering out the candidate word when the number of common characters between the core word and the candidate word is greater than or equal to the number of characters, and the number of identical continuous character sequences between the core word and the candidate word is greater than zero, and the word formed by reversing the character sequence of the core word is the candidate word.

[0077] For example, filtering multiple candidate words for a core word based on named entity type means that if the core word and the candidate word have the same named entity type, the candidate word is retained; otherwise, the candidate word is filtered out.

[0078] For example, the number of shared characters between the core word and the candidate word is used to evaluate the similarity between the two words at the character level. The number of consecutive character sequences that are identical between the core word and the candidate word is used to evaluate whether the two words have overlapping substrings, where a substring is a string containing at least two characters. The word formed by reversing the character sequence of the core word is a candidate word, indicating that the two words are words with the opposite character order.

[0079] In this embodiment, filtering by named entity type ensures consistency between the retained candidate words and the core words in terms of named entity category, thereby improving the relevance and accuracy of the selected synonyms. Furthermore, filtering based on the characters contained in the words relies on character matching, providing effective filtering of candidate words based on literal features. If subsequent filtering based on text similarity is required, filtering based on named entity type and literal characters can provide effective preliminary filtering, improving data processing speed, reducing the complexity of subsequent processing, and thus increasing processing efficiency.

[0080] In some embodiments, text similarity includes sentence similarity. The steps described above for filtering multiple candidate words of a core word based on text similarity may specifically include: for any candidate word of the core word, obtaining the sentence containing the core word and the sentence containing the candidate word, resulting in a first sentence and a second sentence; vectorizing the first and second sentences using multiple pre-trained sentence vector models, obtaining first sentence vectors and second sentence vectors output by each sentence vector model, with different sentence vector models used to vectorize different features of the first sentence and different features of the second sentence; scoring the similarity between the first and second sentences based on the first and second sentence vectors output by each sentence vector model, obtaining multiple first scores; merging the multiple first scores to obtain a second score; and filtering out candidate words if the second score is less than a first predetermined score.

[0081] For example, the sentence containing the core word is taken as the first sentence, and the sentence containing the candidate word is taken as the second sentence. The first sentence is vectorized using multiple pre-trained sentence vector models to obtain the first sentence vector. The second sentence is then vectorized using the same multiple pre-trained sentence vector models to obtain the second sentence vector.

[0082] For example, multiple different sentence vector models have different model types. Sentence vector models are used to convert sentences into corresponding vector forms, thereby capturing the semantic information of sentences.

[0083] As a specific example, sentence vector models can be of at least two types, including but not limited to: paragraph vector (Doc2Vec) models, sentence encoder (Sentence Transformer) models, and sentence-based bidirectional encoder representation (Sentence-BERT) models.

[0084] Among them, the paragraph vector model can generate document-level vector representations, which are suitable for sentence vectorization and can be used to obtain the main idea information of a sentence; the sentence encoder model can use self-attention mechanism and positional encoding to generate sentence vectors, which can be used to obtain the overall semantic information of a sentence, such as macro-level information such as the topic, intention and sentiment of the sentence; the sentence-transformer-based bidirectional encoder representation model is specifically designed to handle sentence-level semantic understanding, generate vector representations of sentences, and can be used to obtain the internal semantic information of a sentence, such as the semantic relationships and interactions between words or phrases in the sentence.

[0085] In some embodiments, the method further includes: acquiring training data, the training data including any corpus data other than behavior logs; and using the training data to train multiple different word vector models and multiple different sentence vector models to obtain multiple pre-trained different word vector models and multiple pre-trained different sentence vector models.

[0086] For example, pre-training refers to training the model using training data before using the word vector model and sentence vector model, so as to obtain multiple different word vector models and multiple different sentence vector models after training.

[0087] For example, the training data is corpus data that does not contain behavior logs. The definition of behavior logs and the source of corpus data can be found in the description of the foregoing embodiments, and will not be repeated here.

[0088] For example, to obtain synonyms of core words in a specific field (such as medicine or law), one can obtain corpus data for that specific field from the corpus data included in the training data. Using the corpus data for that specific field, multiple different word vector models and multiple different sentence vector models can be trained. This allows the pre-trained word vector models and multiple pre-trained sentence vector models to better capture and understand the professional terminology and context of that field, thereby improving the accuracy of obtaining synonyms of core words in that field.

[0089] In this embodiment, multiple different word vector models and sentence vector models are trained using corpus data that does not contain behavior logs. This allows each trained word vector model and sentence vector model to learn rich semantic information, which is beneficial to the accuracy of subsequent vectorization processing results. In related technologies, synonym mining methods usually rely on behavior logs, but behavior logs are often difficult to obtain. However, the word vector models and sentence vector models used in this embodiment are trained using corpus data that does not rely on behavior logs, which can reduce the difficulty of obtaining training data and improve the practicality of the synonym acquisition method.

[0090] In some embodiments, different types of sentence vector models can be trained using historically acquired corpus data, which includes multiple articles or paragraphs. By using these articles or paragraphs to train different types of sentence vector models, each sentence vector model can learn the semantic similarity between sentences from different perspectives, thereby improving the accuracy of similarity determination between sentences.

[0091] In this embodiment, exemplarily, merging multiple first scores can include various methods. When scoring the similarity between the first and second sentences based on the first sentence vector and the second sentence vector, the score is proportional to the similarity between the sentences. For example, when measuring the similarity between sentences using cosine similarity, the calculated cosine value can be directly used as the score.

[0092] It should be understood that the similarity between sentences can also be calculated using any of the following methods, such as Euclidean distance, Manhattan distance, or Jaccard similarity coefficient, and this application does not impose any specific limitations.

[0093] In this embodiment, merging multiple first scores can include various methods.

[0094] For example, a weighted summation method can be used for merging. Specifically, the weight value of any sentence vector model from multiple sentence vector models can be obtained, and this weight value can be set as the weight value of the first score corresponding to that sentence vector model. The weighted sum of the various first scores is then used to obtain the second score. Another example is a simple averaging method. Specifically, the average of the multiple first scores is used to obtain the second score. Yet another example is a truncated averaging method, where the lowest and highest scores are removed from the remaining first scores, and then the average of the remaining first scores is calculated to obtain the second score.

[0095] It should be understood that there are many other ways to merge multiple first scores. The specific method can be chosen according to actual needs, and this application does not impose any specific limitations on the embodiments.

[0096] In this embodiment, when screening multiple candidate words for a core word, leveraging the similarity between their respective sentences helps to understand the semantics of words in context, improving the accuracy of word meaning. Sentence-level similarity analysis allows for more accurate identification of words semantically similar to the core word. Furthermore, in this embodiment, when obtaining the sentence vectors of the sentences to which the core word and candidate words belong, different types of sentence vector models can be used to vectorize both sentences, resulting in multiple sentence vectors for each. Since different types of sentence vector models can capture the distinct semantic features of the sentences to which the core word and candidate words belong, multiple sentence vectors facilitate evaluating the similarity between sentences from different perspectives. A comprehensive understanding of the semantics contained in the sentences makes the sentence similarity-based scoring more accurate.

[0097] In some embodiments, text similarity further includes word similarity. If the second score is greater than or equal to the first predetermined score, the method further includes: vectorizing candidate words using multiple different word vector models to obtain multiple word vectors for candidate words, wherein the word vectors of the core words correspond one-to-one with the word vectors of the candidate words; scoring the similarity between the core words and candidate words based on the one-to-one correspondence between the word vectors of the core words and the word vectors of the candidate words to obtain multiple third scores; merging the multiple third scores to obtain a fourth score; and filtering out candidate words if the fourth score is less than the second predetermined score.

[0098] For example, merging multiple third-party scores can include various methods. When scoring the similarity between core words and candidate words based on their word vectors, the score is proportional to the similarity between the words. Similar to the calculation of sentence similarity, word similarity can be calculated using any of the following methods: cosine similarity, Euclidean distance, Manhattan distance, Jaccard similarity coefficient, etc., without specific limitations in this embodiment.

[0099] As an example, the method for merging multiple third scores can refer to the method for merging multiple first scores described in the above embodiments. This application will not repeat the details in the embodiments.

[0100] As another example, each second score in the above embodiments can be used to indicate the similarity between the sentence containing the core word and the sentence containing a candidate word. In this example, a corresponding score weight value can be set according to each second score. When merging multiple third scores, each third score can be multiplied by the score weight value corresponding to the candidate word to obtain a weighted third score. Then, the weighted third scores are merged to obtain a fourth score.

[0101] In this example, the similarity scores between words are weighted according to the similarity between the sentences to which the words belong. This helps to enhance word-level semantic understanding from the semantic information at the sentence level and improve the accuracy of the filtering results.

[0102] In this embodiment, based on the one-to-one correspondence of core word vectors and candidate word vectors, the similarity between core words and candidate words is scored, and the candidate words are filtered by the score. This can quantitatively evaluate the semantic closeness between core words and candidate words, and efficiently filter out candidate words that are not semantically similar to the core words, thereby improving the quality and accuracy of candidate words.

[0103] According to the synonym acquisition method of this application, multiple word vector models can be used to extract multiple word vectors of the core word. Different word vector models are used to vectorize different features of the core word. Therefore, multiple word vectors of the core word can represent multiple semantic features of the core word. Since multiple semantic features help to cover a wider semantic range, it is beneficial to improve the accuracy of semantic understanding. Therefore, the synonyms of the core word generated based on multiple word vectors have a higher semantic similarity to the core word, which is beneficial to improving the accuracy of the acquired synonyms, thereby obtaining higher quality synonyms. Furthermore, compared with related technologies where manual synonym mining is easily affected by subjective judgment, the method of this application can automatically acquire synonyms from corpus data. While improving the accuracy of the acquired synonyms, it can reduce the time for manual search and screening, thereby helping to reduce labor costs and improve processing efficiency.

[0104] Figure 2 A flowchart of a synonym acquisition method provided for an exemplary embodiment of this application is shown below. Figure 2 As shown, the synonym acquisition method specifically includes the following steps.

[0105] S201, Obtain customer speech data.

[0106] In this step, the customer corpus data includes at least one of arbitrary corpus data sent by the customer and corpus data downloaded from the Internet.

[0107] S202, Identify core words.

[0108] In this step, after acquiring the customer corpus data, it needs to be cleaned to obtain cleaned corpus data. Then, the cleaned corpus data is segmented into words to obtain multiple words. Next, core words are identified through multiple channels. As an example, by calculating the term frequency-inverse document frequency (IF-InVF) of each word in the corpus data, words with IF-InVF values ​​greater than or equal to a predetermined value are identified as core words. Alternatively, external domain-specific dictionaries and domain-specific corpora can be used for identification, and words contained in these dictionaries and corpora can be identified as core words. Furthermore, natural language processing tools can be used to perform part-of-speech tagging and named entity recognition on multiple words in the corpus data, and words with predetermined part-of-speech tags and predetermined named entity types can be identified as core words.

[0109] S203, such as Figure 2 The "core words" shown in the image indicate the core words to be identified.

[0110] S204, train multiple text representation models.

[0111] In this step, to more accurately find potential synonyms, multiple text representation models can be trained using historically acquired customer corpus data. These multiple text representation models include multiple word vector models and multiple sentence vector models. Word vector models are also called word-level text representation models, and sentence vector models are also called sentence-level text representation models. As an example, for the word-level model, a word embedding model can be trained using historically acquired customer corpus data to obtain the word vector model. For the sentence-level model, a domain-specific sentence-level word vector model can be trained based on a large-scale pre-trained model using historically acquired customer data.

[0112] S205, such as Figure 2 As shown in the “Text Representation Model” section, multiple text representation models obtained through training are acquired.

[0113] In this step, multiple text representation models include multiple different word vector models and multiple different sentence vector models. Different word vector models are used to vectorize different features of the core words; different sentence vector models are used to vectorize different features of the first sentence and different features of the second sentence.

[0114] S206, such as Figure 2 As shown in the “candidate words” section, multiple candidate words for the core words are generated by processing the core words based on multiple different word vector models.

[0115] In this step, multiple trained word vector models can be used. For any core word, vector recall is employed to obtain multiple words that are semantically closest to the core word, for example, 20 words. Then, the core word is sequentially combined with the obtained semantically closest words to form a candidate set. That is, the candidate set consists of multiple pairs of words, each pair including two words: one is the core word, and the other is a candidate word of the core word.

[0116] S207, such as Figure 2 As shown in the “Results Integration” section, candidate sets obtained from multiple word vector models are integrated.

[0117] In this step, integration means merging the processing, and filtering is performed during the merging process using at least one of the following methods to obtain all candidate words of the core word after filtering.

[0118] As an example, filtering can be based on the characters contained in the words, i.e., literal features. Filtering is based on the literal overlap between the two words in any pair of words in the candidate set. If one word in the pair is a substring of the other word or the word order of the two words is reversed, it is filtered out; otherwise, it is retained.

[0119] As an example, filtering can be based on named entity type. If the named instances of any two words in the candidate set have the same type, they are retained; otherwise, the candidate words in that pair are filtered out.

[0120] As an example, filtering can be based on context. Specifically, filtering can be performed based on the similarity between the texts containing the two words in any pair of words in the candidate set. That is, a sentence-level text representation model (sentence vector model) is used to score the similarity between the texts containing the two words in the pair. If the score is higher than a certain threshold, the candidate words in the pair are retained; otherwise, the corresponding candidate words are filtered out.

[0121] As an example, filtering can be based on semantic similarity. Specifically, filtering is performed based on the semantic similarity score of the two words in any pair of words in the candidate set. If the score is higher than a certain threshold, the candidate words in that pair of words are retained; otherwise, the corresponding candidate words are filtered out.

[0122] In this step, taking the scoring of the similarity between two words as an example, the similarity scores of the core word and each candidate word can first be normalized; then, the normalized similarity scores are linearly weighted to obtain the weighted score of the core word and each candidate word, also known as the ensemble score; the weighted scores are sorted from largest to smallest, and candidate words with weighted scores higher than a certain preset score threshold are selected to ensure a more accurate candidate set.

[0123] S208, Obtain the integrated candidate words.

[0124] In this step, the integrated candidate words include: all candidate words of the core word after filtering.

[0125] S209, such as Figure 6 As shown in the “Language Model”, the core word and each candidate word are labeled as synonyms using the language model.

[0126] In this step, a language model is used to perform binary classification on each candidate word of the core word, that is, to determine whether each candidate word is a synonym of the corresponding core word. If it is, the candidate word is retained; if not, the candidate word is removed. The retained candidate words are then used as synonyms of the core word, thus obtaining synonyms with higher accuracy.

[0127] S210 outputs the core words and their corresponding synonyms.

[0128] Through the above steps S201-S210, synonyms of each core word in the customer corpus data can be obtained.

[0129] Figure 3A flowchart of the search method provided in the embodiments of this application is shown below. Figure 3 As shown, the search method specifically includes the following steps S310-S330.

[0130] S310 receives a search request, which includes the search terms.

[0131] S320 performs word segmentation on the search content to obtain the search terms in the search content.

[0132] S330, Search for synonyms of the search term from the thesaurus. The thesaurus includes: core words extracted from the corpus data and synonyms of the core words. The synonyms of the core words are words obtained by performing any of the synonym acquisition methods described in the above embodiments on the corpus data.

[0133] S340, retrieve the search term and its synonyms.

[0134] For example, the search content can be one or more search terms, or it can be a sentence or paragraph. By treating the search content as a sequence of characters and performing word segmentation on the search content, the search terms can be obtained.

[0135] In this embodiment, the entity executing the search method can be a terminal device, a single server, a server cluster, a cloud server, etc. These entities can provide search services through the search method.

[0136] It should be understood that, for the sake of convenience and brevity, detailed descriptions of known methods are omitted here. For the specific implementation process of the synonym acquisition method involved in this embodiment, please refer to the implementation process of the corresponding method in the above methods, which will not be repeated here.

[0137] According to the search method of this application embodiment, when conducting a search, synonyms of the search term can be used for retrieval, which helps to increase the number of search results, thereby retrieving more relevant documents and information. Furthermore, since the thesaurus of search terms contains highly accurate synonyms obtained using the aforementioned thesaurus acquisition method, it helps to improve the accuracy of the search results and provide high-quality search results.

[0138] Corresponding to the embodiments of the aforementioned synonym acquisition method, this application also provides embodiments of a synonym acquisition device.

[0139] Figure 4 This is a schematic diagram of a synonym acquisition device according to an embodiment of the present application. The synonym acquisition device is used to execute the synonym acquisition method provided in any of the above embodiments, such as... Figure 4 As shown, the synonym acquisition device includes:

[0140] Extraction module 410 is used to extract core words from the received corpus data. Core words are used to indicate the text features of the corpus data.

[0141] The model processing module 420 is used to vectorize the extracted core words using multiple pre-trained word vector models to obtain multiple word vectors of the core words. Different word vector models are used to vectorize different features of the core words.

[0142] The generation module 430 is used to generate synonyms of the core word based on multiple word vectors.

[0143] In some embodiments, the extraction module 410 may specifically include: performing text cleaning on the corpus data to obtain cleaned corpus data; performing word segmentation on the cleaned corpus data to obtain multiple words in the corpus data; and extracting core words from the multiple words.

[0144] In some embodiments, when the extraction module 410 extracts core words from multiple words, it specifically performs the following steps: calculates the word frequency and inverse document frequency of each word in the corpus data; determines the weight value of the corresponding word based on the word frequency and inverse document frequency; obtains words from multiple words whose weight value is greater than or equal to a predetermined weight threshold to obtain the first core word; obtains words from multiple words that are included in a specific domain corpus to obtain the second core word; obtains the part-of-speech and entity type of each word; obtains words from multiple words whose part-of-speech is a predetermined part-of-speech and whose entity type is a predetermined named entity type to obtain the third core word; and uses the first core word, the second core word, and the third core word as core words extracted from the corpus data.

[0145] In some embodiments, the generation module 430 is specifically used to: generate multiple candidate words that are semantically related to the core word based on multiple word vectors; and obtain synonyms of the core word from the multiple candidate words.

[0146] In some embodiments, when generating multiple candidate words semantically related to a core word based on multiple word vectors, the generation module 430 specifically performs the following: using multiple word vectors to perform semantic similarity-based vector retrieval in a predetermined vector set to obtain a first number of first candidate words, wherein the predetermined vector set includes at least multiple word vectors of each core word; inputting the core word into a predetermined generative model to obtain a second number of second candidate words, wherein the second candidate words are words outside the corpus data and the semantic similarity between the second candidate words and the core word is greater than a predetermined similarity threshold; and generating multiple candidate words semantically related to the core word based on at least one of the first number of first candidate words and the second number of second candidate words.

[0147] In some embodiments, when the generation module 430 is used to obtain a synonym of the core word from multiple candidate words, it is specifically used to: classify the core word and the candidate word using a predetermined language model for any candidate word among the multiple candidate words to obtain a classification result; and confirm the candidate word as a synonym of the core word if the classification result indicates that the core word and the candidate word are synonyms.

[0148] In some embodiments, the synonym acquisition device further includes: a filtering module, configured to filter multiple candidate words based on at least one of named entity type and characters contained in the words; and / or to filter multiple candidate words based on text similarity.

[0149] In some embodiments, when the filtering module filters multiple candidate words based on at least one of named entity type and characters contained in the word, it specifically performs the following filtering process for any candidate word among the multiple candidate words: if the named entity type of the core word is different from the named entity type of the candidate word, the candidate word is filtered out; if the number of common characters between the core word and the candidate word is greater than or equal to the character number threshold, and the number of identical continuous character sequences between the core word and the candidate word is greater than zero, and the word formed by reversing the character sequence of the core word is a candidate word, the candidate word is filtered out.

[0150] In some embodiments, text similarity includes sentence similarity. When the filtering module filters multiple candidate words of a core word based on text similarity, it specifically performs the following steps: for any candidate word of the core word, obtain the sentence containing the core word and the sentence containing the candidate word to obtain a first sentence and a second sentence; use multiple pre-trained sentence vector models to vectorize the first sentence and the second sentence to obtain the first sentence vector and the second sentence vector output by each sentence vector model; score the similarity between the first sentence and the second sentence based on the first sentence vector and the second sentence vector output by each sentence vector model to obtain multiple first scores; merge the multiple first scores to obtain a second score; and filter out candidate words if the second score is less than a first predetermined score.

[0151] In some embodiments, the synonym acquisition device further includes: a model training module, used to acquire training data, the training data including any corpus data other than behavior logs; and to train multiple different word vector models and multiple different sentence vector models using the training data to obtain multiple pre-trained different word vector models and multiple pre-trained different sentence vector models.

[0152] In some embodiments, text similarity further includes word similarity, and the filtering module is further configured to: when the second score is greater than or equal to the first predetermined score, use multiple different word vector models to vectorize the candidate words to obtain multiple word vectors of the candidate words, wherein the word vectors of the core words correspond one-to-one with the word vectors of the candidate words; based on the one-to-one correspondence of the word vectors of the core words and the word vectors of the candidate words, score the similarity between the core words and the candidate words to obtain multiple third scores; merge the multiple third scores to obtain a fourth score; and filter out the candidate words when the fourth score is less than the second predetermined score.

[0153] The synonym acquisition device and the synonym acquisition method provided in this application are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0154] Figure 5 This is a schematic diagram of a search device according to an embodiment of the present application. The search device is used to perform the search method provided in any of the above embodiments, such as... Figure 5 As shown, the search device includes the following modules.

[0155] The receiving module 510 is used to receive search requests, which include search content.

[0156] The word segmentation module 520 is used to segment the search content into words to obtain the search terms in the search content.

[0157] The search module 530 is used to search for synonyms of the search term from the thesaurus. The thesaurus includes: core words extracted from the corpus data and synonyms of the core words. The synonyms of the core words are words obtained by performing any of the above-mentioned synonym acquisition methods on the corpus data.

[0158] The retrieval module 540 is used to retrieve search terms and their synonyms.

[0159] The search device and the search method provided in this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0160] It should be clarified that this application is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the implementation processes of the functions and roles of each module in the above apparatus are specifically described in the implementation processes of the corresponding steps in the above methods, and will not be repeated here.

[0161] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, and the modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0162] Some embodiments of this application also provide an electronic device corresponding to the method provided in the foregoing embodiments, to perform the above-described synonym acquisition method or search method.

[0163] Figure 6 The present invention illustrates a hardware structure diagram of an electronic device according to an exemplary embodiment. The electronic device includes a communication interface 601, a processor 602, a memory 603, and a bus 604. The communication interface 601, processor 602, and memory 603 communicate with each other via the bus 604. The processor 602 can execute the synonym acquisition method or search method described above by reading and executing machine-executable instructions in the memory 603 corresponding to the control logic of the synonym acquisition method or search method. The specific content of this method is described in the above embodiment and will not be repeated here.

[0164] The memory 603 mentioned in this embodiment can be any electronic, magnetic, optical, or other physical storage device, and can contain stored information such as executable instructions, data, etc. Specifically, the memory 603 can be RAM (Random Access Memory), flash memory, storage drive (such as hard disk drive), any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof. Communication between this system network element and at least one other network element is achieved through at least one communication interface 601 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc., can be used.

[0165] Bus 604 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 603 is used to store programs, and processor 602 executes the program after receiving execution instructions.

[0166] Processor 602 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 602 or by instructions in software form. The processor 602 can be a general-purpose processor, including a network processor (NP), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware controls, etc. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.

[0167] The electronic device provided in this application embodiment and the synonym acquisition method or search method provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0168] This application also provides a computer-readable storage medium corresponding to the synonym acquisition or search method provided in the foregoing embodiments. Please refer to... Figure 7 As shown, the computer-readable storage medium shown is an optical disc 70, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the synonym acquisition method or search method provided in any of the aforementioned embodiments.

[0169] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0170] The computer-readable storage medium provided in the above embodiments of this application and the synonym acquisition method or search method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0171] This application also provides a computer program product corresponding to the synonym acquisition method or search method provided in the foregoing embodiments. The computer program product includes a computer program that is executed by a processor to implement the synonym acquisition method or search method provided in the foregoing embodiments.

[0172] The computer program products provided in the above embodiments of this application and the synonym acquisition or search methods provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0173] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0174] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0175] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for obtaining synonyms, characterized in that, The method includes: Core words are extracted from the received corpus data, and the core words are used to indicate the text features of the corpus data; The extracted core words are vectorized using multiple pre-trained word vector models to obtain multiple word vectors for the core words. Different word vector models are used to vectorize different features of the core words. Based on the multiple word vectors, synonyms of the core word are generated.

2. The method according to claim 1, characterized in that, The extraction of core words from the received corpus data includes: The corpus data is cleaned to obtain cleaned corpus data; The cleaned corpus data is segmented to obtain multiple words from the corpus data; Extract the core words from the multiple words.

3. The method according to claim 2, characterized in that, The extraction of core words from the plurality of words includes: Calculate the word frequency and inverse document frequency of each word in the corpus data, determine the weight value of the corresponding word based on the word frequency and the inverse document frequency, and obtain the first core word from the multiple words whose weight value is greater than or equal to a predetermined weight threshold; From the multiple words, words contained in a specific domain corpus are extracted to obtain the second core word; Obtain the part-of-speech and entity type of each word, and from the multiple words, obtain the words whose part-of-speech is a predetermined part-of-speech and whose entity type is a predetermined named entity type to obtain the third core word; The first core word, the second core word, and the third core word are all used as core words extracted from the corpus data.

4. The method according to any one of claims 1-3, characterized in that, The process of generating synonyms for the core word based on the multiple word vectors includes: Based on the multiple word vectors, generate multiple candidate words that are semantically related to the core word; Obtain synonyms of the core word from the plurality of candidate words.

5. The method according to claim 4, characterized in that, The process of generating multiple candidate words semantically related to the core word based on the multiple word vectors includes: Using the multiple word vectors, a vector recall based on semantic similarity is performed in a predetermined vector set to obtain a first number of first candidate words, wherein the predetermined vector set includes at least multiple word vectors of each core word; The core word is input into a predetermined generative model to obtain a second number of second candidate words. The second candidate words are words outside the corpus data, and the semantic similarity between the second candidate words and the core word is greater than a predetermined similarity threshold. Based on at least one of the first number of first candidate words and the second number of second candidate words, generate a plurality of candidate words that are semantically related to the core word.

6. The method according to claim 4, characterized in that, The step of obtaining synonyms of the core word from the plurality of candidate words includes: For any one of the multiple candidate words, the core word and the candidate word are classified using a predetermined language model to obtain a classification result; If the classification result indicates that the core word and the candidate word are synonyms, the candidate word is confirmed as a synonym of the core word.

7. The method according to claim 4, characterized in that, Before obtaining the synonyms of the core word from the plurality of candidate words, the method further includes: The candidate words are filtered based on at least one of the named entity type and the characters contained in the words; and / or, The candidate words are filtered based on text similarity.

8. The method according to claim 7, characterized in that, The filtering of the multiple candidate words based on at least one of named entity type and characters contained in the words includes: For any candidate word among the plurality of candidate words, perform at least one of the following filtering processes: If the named entity type of the core word is different from the named entity type of the candidate word, the candidate word is filtered out. If the number of characters shared between the core word and the candidate word is greater than or equal to the character count threshold, and the number of consecutive character sequences that are the same between the core word and the candidate word is greater than zero, and the word formed by reversing the character sequence of the core word is the candidate word, then the candidate word is filtered out.

9. The method according to claim 7, characterized in that, The text similarity includes sentence similarity, and the filtering of the multiple candidate words based on text similarity includes: For any candidate word of the core word, obtain the sentence containing the core word and the sentence containing the candidate word to obtain the first sentence and the second sentence; The first sentence and the second sentence are vectorized using multiple pre-trained sentence vector models to obtain the first sentence vector and the second sentence vector output by each sentence vector model. Different sentence vector models are used to vectorize different features of the first sentence and different features of the second sentence. Based on the first and second sentence vectors output by each sentence vector model, the similarity between the first and second sentences is scored to obtain multiple first scores; The multiple first scores are merged to obtain the second score; If the second score is less than the first predetermined score, the candidate words are filtered out.

10. The method according to claim 9, characterized in that, The method further includes: Acquire training data, which includes any corpus data other than behavior logs; Multiple different word vector models and multiple different sentence vector models are trained using the training data to obtain multiple pre-trained word vector models and multiple pre-trained sentence vector models.

11. The method according to claim 9, characterized in that, The text similarity also includes word similarity, and the method further includes: when the second score is greater than or equal to the first predetermined score. The candidate words are vectorized using the multiple different word vector models to obtain multiple word vectors for the candidate words, and the word vectors of the core words correspond one-to-one with the word vectors of the candidate words; Based on the word vectors of the core words and the candidate words that correspond one-to-one, the similarity between the core words and the candidate words is scored to obtain multiple third scores; The multiple third scores are merged to obtain the fourth score; If the fourth score is less than the second predetermined score, the candidate words are filtered out.

12. A search method, characterized in that, The method includes: Receive a search request, wherein the search request includes search content; The search content is segmented into words to obtain the search terms in the search content; The search term is searched for in a thesaurus, which includes: core words extracted from corpus data and synonyms of the core words, wherein the synonyms of the core words are words obtained by performing the method of any one of claims 1-11 on the corpus data; The search term and its synonyms are retrieved.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the program to implement the method as described in any one of claims 1-11 or claim 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-11 or claim 12.

15. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method described in any one of claims 1-11 or claim 12.

Images