Method, apparatus, device, medium and program product for generating words

Abstract

Embodiments of the present disclosure relate to methods, devices, equipment, media and program products for generating words, and relate to the fields of data processing technology, text generation technology, large model technology and large language model technology. The method includes determining candidate words to be added to the vocabulary based on multiple documents. The method also includes determining multiple candidate texts for the candidate words based on the candidate words. The method also includes determining the left information entropy and the right information entropy for the candidate words based on the multiple candidate texts. The method also includes determining the candidate words as target words to be added to the vocabulary in response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold. Through this method, the possibility that the candidate words are new words is determined by calculating the left and right information entropies of the candidate words, so that new words in the document can be mined automatically and efficiently, the speed and efficiency of new word mining are improved, and the user experience is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of text processing, and more particularly to methods, devices, apparatuses, media, and program products for generating words. Background Art

[0002] At present, the development of the machine learning industry is becoming faster and faster. More and more different types of machine learning models have emerged and are widely used in different industries and fields. For example, vision-related models can be applied to visual inspection and intelligent driving, and language-related models can be applied to text processing and knowledge question-answering. There are also multimodal machine learning models that integrate various models with different focuses, which can be used in more complex operations that connect multiple different fields.

[0003] With the accelerated development of the machine learning industry, new technologies related to machine learning are emerging one after another, and the application scenarios of machine learning models are becoming more and more numerous. Especially in the field of text processing, the application of machine learning models is becoming more and more extensive. When processing data, there are more and more discussions on how to apply machine learning models more efficiently. For large amounts of data, applying machine learning models when processing will also encounter considerable challenges. Therefore, when faced with massive amounts of data, how to flexibly and efficiently use machine learning models to handle related text processing tasks has become a hot research topic. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, device, medium, and program product for generating words.

[0005] According to a first aspect of the present disclosure, a method for generating words is provided. The method includes determining candidate words to be added to a vocabulary based on multiple documents. The method also includes determining multiple candidate texts for the candidate words based on the candidate words. The method also includes determining left information entropy and right information entropy for the candidate words based on the multiple candidate texts. The method also includes determining the candidate words as target words to be added to the vocabulary in response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold.

[0006] In a second aspect of the present disclosure, a device for generating words is provided. The device includes a candidate word determination module, configured to determine a candidate word to be added to a word library based on multiple documents; a plurality of candidate text determination modules, configured to determine a plurality of candidate texts for the candidate word based on the candidate word; a left information entropy and a right information entropy determination module, configured to determine a left information entropy and a right information entropy for the candidate word based on the plurality of candidate texts; and a target word determination module, configured to determine the candidate word as a target word to be added to the word library in response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold.

[0007] In a third aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a storage device for storing at least one program, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to the first aspect of the present disclosure when executed by a processor.

[0010] It should be understood that the contents described in this content section are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0012] Figure 1 A schematic diagram illustrating an example environment in which the apparatus and / or methods of some embodiments of the present disclosure may be implemented;

[0013] Figure 2 A schematic diagram illustrating an example method for generating words according to some embodiments of the present disclosure is illustrated;

[0014] Figure 3 A schematic diagram illustrating an example of generating candidate words according to some embodiments of the present disclosure is illustrated;

[0015] Figure 4A schematic diagram illustrating an example of a text for generating a word according to some embodiments of the present disclosure;

[0016] Figure 5 A schematic diagram illustrating another example of a text for generating a word according to some embodiments of the present disclosure;

[0017] Figure 6 A schematic diagram illustrating yet another example of a text for generating a word according to some embodiments of the present disclosure;

[0018] Figure 7 A schematic diagram illustrating an example of generating extended information of a word according to some embodiments of the present disclosure;

[0019] Figure 8 A schematic block diagram of an apparatus for generating words according to some embodiments of the present disclosure is illustrated;

[0020] Fig. 9 A schematic block diagram of an example device suitable for implementing various embodiments of the present disclosure is illustrated.

[0021] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0024] For example, when receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0025] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0027] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0028] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0029] New words or professional terms generated in emerging fields cannot be found in traditional lexicons, which brings huge challenges to multilingual text processing business scenarios such as information retrieval, recommendation systems, intelligent customer service, marketing, etc. Traditional new word mining methods, such as statistical-based methods, do not handle low-frequency words and polysemous words well and cannot meet the needs of new word mining. Therefore, how to automatically and efficiently mine new words in emerging fields has become a technical problem that needs to be solved urgently.

[0030] In traditional solutions, the following methods can be used to realize the recognition of new words. One method is a statistical method, which mainly discovers new words by calculating the statistical characteristics of words, such as word frequency, mutual information (MI), left and right entropy, etc. Another method is a machine learning method, which mainly discovers new words by training machine learning models, such as decision trees, support vector machines, neural networks, etc. Another method is a rule-based method, which mainly discovers new words by defining a series of rules, such as word length, frequency, position, etc. Although the above-mentioned methods can solve the problem of new word recognition to a certain extent, there are still some defects. For example, although the statistical method is simple and easy to implement, it has poor processing effect on low-frequency words and long words, is sensitive to noise, cannot process semantic information, relies on large-scale corpus, has fuzzy boundaries, and cannot handle polysemy and ambiguity. In addition, although the machine learning method can handle complex features and nonlinear relationships, it requires a large amount of labeled data and the training process is time-consuming. In addition, although the rule-based method is simple and easy to implement, the definition and adjustment of rules requires a lot of manual participation, and the effect of discovering new words is limited, and the user experience is reduced.

[0031] At least to solve the above and other potential problems, an embodiment of the present disclosure proposes a method for generating words. In this method, a computing device can first determine candidate words to be added to the vocabulary based on multiple documents. After determining the candidate words, the computing device can use the determined candidate words to determine multiple candidate texts for the candidate words. Then, after determining multiple candidate texts, the computing device can further determine the left information entropy and right information entropy for the candidate words. Finally, if the left information entropy is greater than a first predetermined threshold and the right information entropy is greater than a second predetermined threshold, the candidate word is determined as a target word to be added to the vocabulary. Through this method, the possibility that the candidate word is a new word is determined by calculating the left and right information entropies of the candidate word, so that new words in the document can be automatically and efficiently mined, the speed and efficiency of new word mining are improved, and the user experience is improved.

[0032] The embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. Figure 1 An example environment in which the device and / or method of the embodiments of the present disclosure may be implemented is shown. In environment 100, computing device 102 is used to process a large amount of data corresponding to a plurality of documents to determine target words that can be added to a vocabulary.

[0033] Examples of computing device 102 include, but are not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), a media player, etc.), a multiprocessor system, consumer electronics, a minicomputer, a mainframe computer, a distributed computing environment including any of the above systems or devices, and the like.

[0034] It is understandable that the present application is only described in combination with relevant embodiments, and the protection scope of the present application is not limited by the relevant embodiments. Any technical solutions in other applications that fall within the protection scope of the present application should be protected.

[0035] like Figure 1 As shown, in Figure 1 In the environment 100, the computing device 102 first determines the candidate words 106 to be added to the vocabulary based on the multiple documents 104. In one example, the multiple documents 104 are original documents that have not been preprocessed. In another example, the multiple documents 104 can be obtained after preprocessing the multiple original documents. The preprocessing process includes one or more of data cleaning, text normalization or word segmentation operations for the multiple original documents. In some embodiments, the candidate words are determined by inputting the multiple documents 104 into a first machine learning model.

[0036] Figure 1 It is shown that a candidate word is determined from multiple documents 104, which is only an example and not a specific limitation of the present disclosure. The computing device 102 can determine multiple candidate words from multiple documents 104, where each candidate word is processed in a similar manner to the candidate word 106.

[0037] Next, after the candidate words 106 are determined, the computing device 102 continues to generate a plurality of candidate texts 108 according to the determined candidate words 106. For example, the plurality of candidate texts 108 for the plurality of candidate words 106 are generated by processing using a second machine learning model.

[0038] Additionally, the first machine learning model and the second machine learning model may be language models. In one example, the first machine learning model and the second machine learning model are the same language model (e.g., BERT model, LLaMA 2 model, etc.). In another example, the first machine learning model and the second machine learning model may be different language models. In yet another example, the first machine learning model and the second machine learning model may be language models fine-tuned using word data and text data, respectively. It is to be understood that the first machine learning model and the second machine learning model may be any language model or other model that can implement the present application, and the present application is not limited thereto.

[0039] Then, after determining a plurality of candidate texts 108 for the candidate word 106 , the information entropy of the candidate word 106 , that is, the left information entropy 110 and the right information entropy 112 of the candidate word 106 in the plurality of candidate texts 108 , may be further calculated.

[0040] The left information entropy 110 and the right information entropy 112 respectively represent the diversity of the left context and the diversity of the right context of the candidate word 106 in the multiple candidate texts 108. In addition, the higher the values ​​of the left information entropy 110 and the right information entropy 112, the higher the independence and freedom of the candidate word 106 in different contexts, and the more likely it is to be a new word or an important word.

[0041] Finally, after the values ​​of the left information entropy 110 and the right information entropy 112 are calculated, a judgment can be made based on the left information entropy 110 and the first predetermined threshold value and the right information entropy 112 and the second predetermined threshold value. Additionally, when the values ​​of the left information entropy 110 and the right information entropy 112 are respectively greater than the first predetermined threshold value and the second predetermined threshold value, the candidate word 106 is determined as the target word 114.

[0042] After the candidate word 106 is determined as the target word 114, the target word 114 will be added to the vocabulary. Additionally, the target word needs to be further confirmed, and then the target word is added to the vocabulary according to the result of the confirmation. Whenever a target word 114 is added to the vocabulary, the vocabulary will be updated accordingly. Additionally, the quantized representation corresponding to the target word 114 will also be stored in the corpus corresponding to the vocabulary.

[0043] Through this method, the possibility of a candidate word being a new word is determined by calculating the left and right information entropy of the candidate word, so that new words in the document can be automatically and efficiently mined, the speed and efficiency of new word mining is improved, and the user experience is improved.

[0044] Combined with the above Figure 1 A schematic diagram of an example environment in which the devices and / or methods of some embodiments of the present disclosure may be implemented is described below. Figure 2 A schematic diagram depicting an example method 200 for generating a term according to some embodiments of the present disclosure. The example method may be Figure 1 Executed by computing device 102 or any suitable computing device.

[0045] like Figure 2As shown, in the example method 200, at block 202, the computing device 102 determines candidate words to be added to the vocabulary based on multiple documents. In order to obtain possible new words, a large number of documents need to be mined. Therefore, the computing device 102 needs to pre-determine candidate words that may be new words from these documents.

[0046] In some embodiments, the multiple documents are documents obtained after preprocessing the multiple original documents. The preprocessing of the multiple original documents includes operations such as data cleaning, text normalization or word segmentation. In one example, when the multiple original documents are directly obtained from the web page, the multiple original documents need to be data cleaned, and the hypertext markup language in the original documents can be deleted. In another example, when the text in the multiple original documents is not in a standardized format or has special characters, etc., the text in the multiple original documents can be normalized. In another example, a model dedicated to processing text segmentation, such as a hidden Markov model (HMM), conditional random fields (CRFs), etc., can be used to segment the text in the multiple original documents to facilitate the subsequent machine learning model to identify and determine the candidate words.

[0047] In other embodiments, the left information entropy and the right information entropy of multiple candidate words in multiple original documents can be directly calculated, and then the left information entropy and the right information entropy of multiple entities are subjected to mutual information screening, and then the remaining entities after screening are applied to the first machine learning model to determine the candidate words for the multiple original documents.

[0048] The computing device 102 may process multiple documents to determine entities in the multiple documents. After determining the entities in the multiple documents, further obtain relevant contexts corresponding to the entities from a vocabulary or corpus based on the determined entities. The vocabulary stores a large number of entries, each of which includes a word and a corresponding sentence. The corpus stores a large number of collected documents.

[0049] In some embodiments, after determining the entity for multiple documents, the computing device 102 searches for reference words corresponding to the entity in the vocabulary. If the reference words corresponding to the entity are found in the vocabulary, the description information or entries corresponding to the reference words in the vocabulary are determined as the relevant context associated with the entity. If the reference words corresponding to the entity are not found in the vocabulary, it is necessary to further search in the corpus.

[0050] For example, the computing device first determines a quantitative representation of an entity, and then obtains a sentence corresponding to the entity from a corpus as a relevant context corresponding to the entity based on the determined quantitative representation of the entity.

[0051] In some embodiments, the computing device uses a first machine learning model to determine a quantitative representation of an entity, and the text data in the corpus is stored in the form of a quantitative representation. The computing device can use an edit distance algorithm such as the Jaro-winkler algorithm or the BM-25 (Best Matching 25) retrieval algorithm to retrieve and match from the corpus to determine the quantitative representation of the sentence corresponding to the quantitative representation of the entity. Finally, the determined sentence is determined as the relevant context corresponding to the entity.

[0052] In some embodiments, computing device 102 applies multiple documents and relevant contexts corresponding to entities in multiple documents to a first machine learning model, such as a Bert language model or a large language model, to determine candidate words to be added to the vocabulary. In some embodiments, a mapping relationship between documents, contexts, and entities is pre-determined, and the mapping relationship is used to find candidate words corresponding to multiple documents and relevant contexts.

[0053] In some embodiments, there may be only one candidate word determined. In other embodiments, there may be multiple candidate words determined.

[0054] Next, at box 204, multiple candidate texts for the candidate word are determined based on the candidate word. In order to more accurately determine whether a candidate word can become a new word, it is also necessary to generate multiple candidate texts for the candidate word for detecting the independence and degree of freedom of the candidate word. In some embodiments, the computing device 102 first uses a second machine learning model to generate multiple candidate texts for the candidate word. Among them, the number of multiple candidate texts is indefinite and can be defined by the user. For example, the second machine learning model can be a language model, such as a large language model. In some embodiments, the multiple candidate texts are obtained by other means, such as searching from a corpus.

[0055] In some embodiments, the position of the candidate word in the plurality of candidate texts may be a prefix of the candidate text or a suffix of the candidate text. Additionally, the position of the candidate word in the plurality of candidate texts may be any position in the middle of the candidate text.

[0056] In some embodiments, the number of multiple candidate texts can be large. Since the capabilities of the machine learning model are limited, the number of different candidate texts generated for candidate words is limited. When the machine learning model cannot generate more different candidate texts, it will repeatedly generate repeated candidate texts.

[0057] Then, at block 206, based on the plurality of candidate texts, the left information entropy and the right information entropy for the candidate word are determined. After obtaining the plurality of candidate texts, the information entropy of the candidate word needs to be calculated to detect whether the candidate word can be used as a new word.

[0058] Additionally, as described above, the generation of multiple candidate texts for candidate words using a machine learning model is limited. When a large number of multiple candidate texts are generated, the machine learning model may generate a large number of repeated candidate texts. For example, the number of multiple candidate texts generated for candidate words can be set to be greater than or equal to 100. For a large number of multiple candidate texts, repeated candidate texts can be further eliminated to facilitate more accurate calculation results of left information entropy and right information entropy.

[0059] In some embodiments, after a large number of candidate texts are generated, the text similarity between the multiple candidate texts can be determined, and the range of the text similarity is [0, 1]. In one example, the Jaro-winkler algorithm or the BM-25 (Best Matching 25) retrieval algorithm can be used to calculate the text similarity between the multiple candidate texts.

[0060] Then, the candidate texts whose calculated text similarity is greater than the threshold are deleted from the multiple candidate texts, and the remaining candidate texts whose text similarity is less than or equal to the threshold are retained. For example, the candidate texts whose text similarity is greater than 0.8 are deleted from the multiple candidate texts, and the remaining candidate texts whose text similarity is less than or equal to 0.8 are retained.

[0061] Additionally, only the candidate texts with a text similarity of 1 may be deleted from the multiple candidate texts, that is, multiple identical repeated candidate texts are deleted from the multiple candidate texts, and only any one of the multiple repeated candidate texts is selected and retained.

[0062] After the deletion of the generated multiple candidate texts is completed, the computing device may calculate the left information entropy and the right information entropy for the candidate words according to the remaining candidate texts in the multiple candidate texts.

[0063] Finally, at block 208 , in response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold, the candidate word is determined as a target word to be added to the vocabulary.

[0064] After the computing device calculates and determines the left information entropy and the right information entropy for a candidate word based on the candidate texts among multiple candidate texts, it is necessary to further determine whether the candidate word can be determined as a target word to be added to the vocabulary based on the value of the left information entropy and the size of the first predetermined threshold and the value of the right information entropy and the size of the second predetermined threshold.

[0065] In some embodiments, when the value of the left information entropy and the value of the right information entropy are respectively greater than the first predetermined threshold and the second predetermined threshold, the candidate word is determined as the target word. In one example, the first predetermined threshold and the second predetermined threshold are the same. In another example, the first predetermined threshold and the second predetermined threshold are different.

[0066] In some embodiments, after the target word is determined, it is further confirmed whether the target word can be added to the word library. For example, the validity of the target word is further confirmed. Only when the determined target word is determined to be valid, the target word is a new word that has been discovered and can be added to the word library.

[0067] In one example, the validity of the target word can be manually confirmed. In another example, the language model used to determine the candidate words can be fine-tuned, and then the fine-tuned language model can be used to confirm the validity of the determined target word. In yet another example, time series analysis can also be performed by detecting the time series of the target word in the document to detect the abnormality of the target word. For example, when the time frequency of the target word in the document does not conform to the normal distribution that conventional words conform to, the target word can be confirmed as valid.

[0068] In some embodiments, after the target word is determined, the context for the target word can be further determined from the multiple documents. The determined target word and the context for the target word are then saved in the form of entries, and the entries are saved in the word library.

[0069] Additionally, the computing device may also confirm the application scenario and attribute of the target word from the word library or corpus according to the target word and the context of the target word. For example, when the target word is determined to be "limited compensation" and it is determined that the target word appears in an e-commerce scenario, the attribute of the target word "limited compensation" may be further determined to be "activity".

[0070] After the application scenario and attributes for the target word are determined, the target word, the context corresponding to the target word, the application scenario corresponding to the target word, and the attributes corresponding to the target word may be stored in the word library in the form of entries.

[0071] In some embodiments, after determining the entry for the candidate word, the computing device can further use the determined entry to search from the word library or corpus to find synonyms with similar meanings to the target word or antonyms with opposite meanings. In addition, the application scenarios and attributes of the synonyms or antonyms determined for the target word can also be used to confirm and update the application scenarios and attributes of the target word.

[0072] In some embodiments, when the target word is determined and determined to be valid, it can be stored in the word library, and the extended information related to the target word can be supplemented after being saved in the word library. For example, the extended information for the target word is stored in the word library or corpus in association through Internet search.

[0073] In some embodiments, when a user is searching, if a target word is included in the user's query input, the target word may be highlighted, for example, the target word may be underlined or bolded.

[0074] Then, when a cursor hovers over the target word, the extended information for the target word is directly displayed to the user. Additionally, the entry, application scenario or attribute for the target word can be displayed to the user.

[0075] Through this method, a machine learning model is used to generate candidate words, and the possibility that the candidate words are new words is determined by calculating the left and right information entropy of the candidate words, and the validity of the new words can be further confirmed, so that new words in documents can be automatically and efficiently mined, and can be widely used in multilingual text processing business scenarios, improving user experience.

[0076] Combined with the above Figure 2 A schematic diagram of an example method for generating a word according to some disclosed embodiments is described. Figure 3 A schematic diagram for generating an example for candidate terms according to some embodiments of the present disclosure is described.

[0077] like Figure 3 As shown in example 300 , after the computing device 102 receives a plurality of texts, the computing device starts a processing flow for the input texts.

[0078] After the computing device 102 receives a document among multiple documents, it will display the determined target words and show them to the user in the corresponding output box. In one example, after the computing device receives the document "What is the vv in the last 7 days" in the document box 302, it will directly determine the target word "vv" for the document and show it to the user in the output box 304. Among them, the meaning of "vv" can refer to "video view", so the meaning of the text refers to the sum of the number of times a video or other multimodal content is opened within a statistical period (such as 7 days). At this time, the candidate word for the new word determined by the computing device 102 is "vv". In another example, the computing device can also determine the candidate word "Baibu" for the document after receiving the document "Baibu products in the last 7 days" in the document box 306, and then show it to the user in the output box 308.

[0079] In some embodiments, before the computing device determines the candidate words for multiple documents, it will preprocess multiple original documents of the multiple documents. The preprocessing of multiple original documents is generally to facilitate the subsequent more efficient and effective application of text data in a machine learning model and improve the efficiency of generating candidate words and candidate texts.

[0080] During the preprocessing of multiple original documents, it is necessary to clean the data of the documents. In one example, when the document is crawled from a web page, it is necessary to remove the HyperText Markup Language (HTML) contained in the document. In another example, it is also necessary to remove special characters or emojis in the document, such as removing punctuation marks, emojis, etc. In yet another example, it is also necessary to remove stop words in the document, such as common words like "de", "le", "zai", etc. In still another example, it is also necessary to remove blank lines and extra spaces in the document to ensure the regular format of the document.

[0081] During the preprocessing of multiple original documents, text normalization will also be performed on the multiple original documents. The process of text normalization can remove the extra spaces in the input text, and when there are English letters in the input text, it can convert the case of the English letters. For example, if the input text is "x brand's official store", after text normalization, the processed result is "x brand's official store".

[0082] In some embodiments, in addition to removing spaces and converting uppercase and lowercase letters, punctuation marks in the input text can also be standardized, such as processing full-width and half-width conversions, and processing the mixing of Chinese and English punctuation marks. It is also possible to process the conversion between Arabic numerals and Chinese numerals, and process the standardization of special numerical forms such as scientific notation and fractions. In other embodiments, it is also possible to convert simplified and traditional Chinese characters, unify variant forms, process special characters, etc.

[0083] In the process of preprocessing the multiple original documents, word segmentation is also performed. In this process, the multiple original documents are segmented, and word sequences for the multiple original documents are generated according to the results of the word segmentation.

[0084] For example, if one of the original documents among the multiple original documents is "Beijing's weather", during the word segmentation process, the original document will be divided into two words, "Beijing" and "weather", and the separated words will be sorted in order, thereby generating a word sequence for the original document. In addition, the way to sort the words is defined by the user, for example, the words can be sorted in the order in which they appear in the original document, or they can be sorted by the first letter of the pinyin.

[0085] Additionally, during the word segmentation process, the original document may be segmented according to the previous word segmentation history records, which include information such as the user's input habits, frequently input words, and query fields.

[0086] Finally, after the target words are determined, the display information corresponding to the determined target words can also be displayed, and a group of target words can be processed in real time, such as performing a real-time online search, and the extended information for the target words can be displayed to the user.

[0087] Combined with the above Figure 3 A schematic diagram of an example for generating candidate words according to some disclosed embodiments is described. Figure 4 A schematic diagram describing an example of a text for generating words according to some embodiments of the present disclosure.

[0088] like Figure 4 As shown, in example 400, combined with the previous example 300, after the candidate word is determined, the computing device further utilizes the language model to generate multiple candidate texts for the candidate word (for example, 20 candidate texts, the number of candidate texts can be defined by the user).

[0089] Additionally, a language model may be used to generate multiple candidate texts for prefixes and suffixes of a candidate word. Generating multiple prefix candidate texts and multiple suffix candidate texts may increase the diversity of the multiple candidate texts.

[0090] For example, after inputting the candidate word "dual - end" into the input box 402, the language model generates 20 texts for "dual - end" in prefix and suffix forms according to pre - set rules. Then, according to the prefix and suffix forms, the corresponding number of generated texts are respectively displayed in the output box 404 for the user to view.

[0091] Among them, the texts in prefix form generated for "dual - end" include: "efficient dual - end", "intelligent dual - end", "portable dual - end", "multi - functional dual - end", "high - speed dual - end", "reliable dual - end", "advanced dual - end", "economic dual - end", "environment - friendly dual - end", "safe dual - end".

[0092] The texts in suffix form generated for "dual - end" include: "dual - end connection", "dual - end interface", "dual - end transmission", "dual - end device", "dual - end system", "dual - end communication", "dual - end control", "dual - end application", "dual - end design", "dual - end solution".

[0093] As described above in conjunction with Figure 4 a schematic diagram of an example for generating texts of words according to some disclosed embodiments is described. Below, in conjunction with Figure 5 a schematic diagram of another example for generating texts of words according to some embodiments of the present disclosure is described.

[0094] As Figure 5 shown, in Example 500, in combination with the previous Example 300, after inputting the candidate word "hundred - supplement" into the input box 502, the language model generates 20 texts for "hundred - supplement" in prefix and suffix forms according to pre - set rules. Then, according to the prefix and suffix forms, the corresponding number of generated texts are respectively displayed in the output box 504 for the user to view.

[0095] Among them, the texts in prefix form generated for "hundred - supplement" include: "preferential hundred - supplement", "planned hundred - supplement", "event hundred - supplement", "special - zone hundred - supplement", "recommended hundred - supplement", "selected hundred - supplement", "special - offer hundred - supplement", "exclusive hundred - supplement", "welfare hundred - supplement", "time - limited hundred - supplement".

[0096] The texts in suffix form generated for "hundred - supplement" include: "hundred - supplement rush purchase", "hundred - supplement hot sale", "hundred - supplement goods", "hundred - supplement discount", "hundred - supplement flash sale", "hundred - supplement carnival", "hundred - supplement festival", "hundred - supplement grand ceremony", "hundred - supplement big promotion", "hundred - supplement selection".

[0097] After generating 20 texts for "hundred - supplement", the computing device further processes the generated texts, for example, by calculating the left - right information entropy of the generated texts to determine the candidate word as the target word.

[0098] Among them, the left information entropy in the left and right information entropy is used to measure the context diversity on the left side of the candidate word, while the right information entropy in the left and right information entropy is used to measure the context diversity on the right side of the candidate word. The magnitude of the left and right information entropy metrics can be used to judge the freedom and independence of the candidate word in the generated text.

[0099] Generally speaking, a high left and right information entropy value usually means that the candidate word appears in different contexts, so it has high independence and freedom and is more likely to be the newly discovered word.

[0100] Generally, before calculating the left and right information entropy, it is necessary to first use the language model to generate a large number (for example, 100) of texts, and then exclude the generated texts before calculating the left and right information entropy, and the obtained results will be more accurate. The following will be combined with Figure 6 for a detailed description.

[0101] As described above in combination with Figure 5 a schematic diagram of another example of the text for generating words according to some embodiments of the present disclosure is described. The following will be combined with Figure 6 to describe a schematic diagram of yet another example of the text for generating words according to some embodiments of the present disclosure.

[0102] As Figure 6 shown, in Example 600, after determining the candidate word "Baibu", "Baibu" is input into the input box 602, and the language model is used to further generate 100 texts to be displayed in the output box 604. The purpose of generating a large number of texts is to generate texts in different contexts as much as possible to make the calculation result of the left and right information entropy more accurate.

[0103] However, since the language model cannot generate a large number of non-repetitive texts, when a large number of texts need to be generated, there will be a situation of generating duplicate texts. In order to avoid the influence of duplicate texts on the calculation of the left and right information entropy value, it is necessary to first剔除 the similar or identical texts in the generated duplicate texts. For example, use the Jaro-winkler algorithm based on the edit distance to剔除 the generated texts with extremely high or identical text similarity. Additionally, one of the multiple similar or identical texts will also be retained for subsequent calculation of the left and right information entropy.

[0104] Then, after the similar or identical texts are剔除, the left and right information entropy is calculated for the remaining texts. Among them, the calculation formula for the left information entropy of the candidate word is as shown in the following formula (1):

[0105] (1)

[0106] Among them, Denotes the set of all distinct characters that appear on the left side, and denotes the probability that the character appears on the left side.

[0107] The calculation formula for the right information entropy of a candidate word is shown in the following formula (2):

[0108] (2)

[0109] Wherein, denotes the set of all distinct characters that appear on the right side, and denotes the probability that the character appears on the right side.

[0110] After the calculation of the left and right information entropy is completed, when the values of the left information entropy and the right information entropy are both higher than the corresponding predetermined thresholds, the candidate word can be determined as the target word. Among them, the predetermined threshold can be defined by the user himself, and this application does not limit it here.

[0111] For example, in Example 600, after 100 texts are generated for the candidate word "Baibu" using a language model, two texts with a large number of repetitions, "Baibu Hot Sale" and "Baibu Rush Purchase", appear. In this case, it is necessary to eliminate the repeated "Baibu Hot Sale" and "Baibu Rush Purchase" according to the text similarity, and then calculate the left and right information entropy based on the remaining texts after elimination.

[0112] In this way, the values of the calculated left and right information entropy can be made more accurate, and the possibility that the determined target word is a new word is higher.

[0113] The above combines Figure 6 to describe a schematic diagram of another example of generating text using a machine learning model for generating words according to some embodiments of the present disclosure. The following combines Figure 7 to describe a schematic diagram of an example of extended information for generating words according to some embodiments of the present disclosure.

[0114] As Figure 7 shown, in Example 700, when the user enters the document "GMV of x brand in the last 7 days" in the search box 710, the input word "x brand" 708 in the document is recognized as the target word and is highlighted with an underline.

[0115] It can be seen that the extended information 704 “x brand flagship store” and the extended information 706 “x brand” related to “x brand” are also displayed in the box 702 of suggestions related to “x brand”.

[0116] Additionally, the extended information also displays the attributes corresponding to each piece of extended information. For example, the attribute of extended information 704 is determined to be a store, and the attribute of extended information 706 is determined to be a brand.

[0117] In some embodiments, the selection of the extended information can be determined by moving the cursor. In other embodiments, the selection of the extended information can be switched by pressing the "Tab" key on the keyboard.

[0118] In some embodiments, the extended information may be multiple pieces. In other embodiments, the extended information may be only one piece.

[0119] like Figure 8 As shown, the device 800 includes a candidate word determination module 802, which is configured to determine a candidate word to be added to the vocabulary based on multiple documents; a multiple candidate text determination module 804, which is configured to determine multiple candidate texts for the candidate word based on the candidate word; a left information entropy and right information entropy determination module 806, which is configured to determine the left information entropy and right information entropy for the candidate word based on multiple candidate texts; and a target word determination module 808, which is configured to determine the candidate word as a target word to be added to the vocabulary in response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold.

[0120] In some embodiments, the candidate word determination module 802 includes: an entity determination module, configured to determine entities in multiple documents; a relevant context acquisition module, configured to acquire relevant context for the entity based on at least one of a vocabulary and a text corpus; and a candidate word determination module, configured to determine candidate words to be added to the vocabulary based on multiple documents and relevant context.

[0121] In some embodiments, the relevant context acquisition module includes: a reference word determination module, configured to search for a reference word corresponding to an entity in a vocabulary; and a description information acquisition module for the reference word, configured to acquire description information for the reference word in the vocabulary as relevant context in response to the presence of a reference word corresponding to the entity in the vocabulary.

[0122] In some embodiments, the relevant context acquisition module also includes: a quantitative representation determination module, configured to determine the quantitative representation of the entity in response to the absence of a reference word corresponding to the entity in the vocabulary; and a sentence acquisition module corresponding to the entity, configured to acquire the sentence corresponding to the entity as the relevant context from the text corpus based on the quantitative representation.

[0123] In some embodiments, the candidate term determination module includes: a first machine learning model application module, configured to determine candidate terms for multiple documents by applying the multiple documents and related contexts to a first machine learning model.

[0124] In some embodiments, the multiple candidate text determination module 804 includes: a second machine learning model application module, configured to generate multiple candidate texts for the candidate words by applying the candidate words to a second machine learning model.

[0125] In some embodiments, the left information entropy and right information entropy determination module 806 includes: a candidate text deletion module with text similarity greater than a threshold similarity, which is configured to delete candidate texts with text similarity greater than a threshold similarity from multiple candidate texts based on the similarity between multiple candidate texts; and a left information entropy and right information entropy determination module, which is configured to determine the left information entropy and right information entropy for the candidate words based on the remaining candidate texts in the multiple candidate texts.

[0126] In some embodiments, the device 800 also includes: a target word addition confirmation module, configured to confirm whether the target word can be added to the vocabulary; a target word context determination module, configured to determine the context for the target word in response to the target word being confirmed to be added to the vocabulary; a target word entry determination module, configured to determine the entry for the target word based on the context of the target word; and an entry saving module, configured to save the entry in the vocabulary.

[0127] In some embodiments, the apparatus 800 further includes: a module for acquiring at least one of synonyms and antonyms, configured to acquire at least one of synonyms and antonyms associated with the target word based on the entry.

[0128] In some embodiments, the apparatus 800 further includes: a plurality of original document acquisition modules configured to acquire a plurality of original documents; and a plurality of document generation modules configured to generate a plurality of documents by pre-processing the plurality of original documents.

[0129] In some embodiments, the preprocessing includes at least one of the following: data cleaning, text normalization, or word segmentation operation.

[0130] In some embodiments, the device 800 also includes: a user query input acceptance module, configured to receive the user's query input; a target word highlighting module, configured to highlight the target word in the query input in response to the query input including the target word; and an extended information display module for the target word, configured to display extended information for the target word in response to receiving a cursor hovering operation on the target word.

[0131] Fig. 9 A schematic block diagram of an example device 900 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 102 in the embodiment can be implemented by using the device 900. As shown in the figure, the device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0132] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0133] The various processes and processing described above, such as method 200 and examples 300 to 700, may be performed by the processing unit 901. For example, in some embodiments, method 200 and examples 300 to 700 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of the example method 200 and examples 300 to 700 described above may be performed.

[0134] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0135] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive lists) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage medium used herein is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0136] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0137] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0138] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0141] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0142] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a word, comprising: Based on multiple documents, determine candidate words to be added to the vocabulary; Based on the candidate words, determining a plurality of candidate texts for the candidate words; Based on the multiple candidate texts, determining the left information entropy and the right information entropy for the candidate words; as well as In response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold, determining the candidate word as a target word to be added to the vocabulary; The candidate words to be added to the vocabulary include: determining entities in the plurality of documents; Based on at least one of the vocabulary and the text corpus, obtaining relevant context for the entity; and Determining the candidate words to be added to the vocabulary based on the multiple documents and the relevant contexts; Wherein obtaining the relevant context for the entity comprises: searching the vocabulary for a reference word corresponding to the entity; and In response to the reference word not existing in the vocabulary and corresponding to the entity, determining a quantified representation of the entity; and Based on the quantified representation, a sentence corresponding to the entity is obtained from the text corpus as the relevant context.

2. The method according to claim 1, wherein obtaining relevant context for the entity further comprises: In response to the existence of a reference word corresponding to the entity in the vocabulary, description information for the reference word in the vocabulary is obtained as the relevant context.

3. The method according to claim 1, wherein determining candidate words to be added to the vocabulary based on the plurality of documents and the relevant context comprises: The candidate terms for the plurality of documents are determined by applying the plurality of documents and the relevant context to a first machine learning model.

4. The method according to claim 1, wherein determining a plurality of candidate texts for the candidate term comprises: The plurality of candidate texts for the candidate words are generated by applying the candidate words to a second machine learning model.

5. The method according to claim 1, wherein determining the left information entropy and the right information entropy for the candidate word comprises: Based on the similarities between the multiple candidate texts, delete the candidate texts whose text similarity is greater than a threshold similarity from the multiple candidate texts; as well as Based on the remaining candidate texts in the plurality of candidate texts, the left information entropy and the right information entropy for the candidate word are determined.

6. The method according to claim 1, further comprising: Confirming whether the target word can be added to the word library; In response to the target word being confirmed to be added to the vocabulary, determining a context for the target word; Determining a vocabulary entry for the target word based on the context of the target word; as well as The entry is saved in the vocabulary.

7. The method according to claim 6, further comprising: Based on the entry, at least one of a synonym and an antonym associated with the target word is obtained.

8. The method according to claim 1, further comprising: Obtain multiple original documents; as well as The plurality of documents are generated by pre-processing the plurality of original documents.

9. The method of claim 8, wherein the preprocessing comprises at least one of the following: Data cleaning, text normalization or word segmentation operations.

10. The method according to claim 1, further comprising: Receive query input from the user; In response to the query input including the target word, highlighting the target word in the query input; as well as In response to receiving a hover operation of the cursor on the target word, extended information for the target word is displayed.

11. A device for generating a word, comprising: A candidate word determination module is configured to determine candidate words to be added to the vocabulary based on multiple documents; A plurality of candidate text determination modules, configured to determine a plurality of candidate texts for the candidate words based on the candidate words; A left information entropy and a right information entropy determination module, configured to determine the left information entropy and the right information entropy for the candidate words based on the multiple candidate texts; as well as a target word determination module, configured to determine the candidate word as a target word to be added to the vocabulary in response to the left information entropy being greater than a first predetermined threshold and the right information entropy being greater than a second predetermined threshold; The candidate word determination module comprises: An entity determination module configured to determine entities in the plurality of documents; A relevant context acquisition module configured to acquire relevant context for the entity based on at least one of the vocabulary and the text corpus; and a candidate word determination module, configured to determine the candidate words to be added to the vocabulary based on the multiple documents and the relevant contexts; The relevant context acquisition module includes: a reference word determination module configured to search the word library for a reference word corresponding to the entity; and a quantitative representation determination module configured to determine a quantitative representation of the entity in response to the absence of the reference word corresponding to the entity in the vocabulary; and The sentence acquisition module is configured to acquire, based on the quantified representation, a sentence corresponding to the entity from the text corpus as the relevant context.

12. An electronic device comprising: at least one processor; as well as A storage device, used for storing at least one program, when the at least one program is executed by the at least one processor, so that the at least one processor implements the method according to any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images