Dictionary updating method and device, terminal equipment and computer readable storage medium

Abstract

The application discloses a dictionary updating method and device, a terminal equipment and a computer readable storage medium. The method comprises the following steps: obtaining a to-be-processed vocabulary; performing semantic analysis on the to-be-processed vocabulary to determine a semantic analysis result of the to-be-processed vocabulary; determining a target vocabulary in the to-be-processed vocabulary according to the semantic analysis result; and updating a dictionary according to the target vocabulary. The method can introduce the semantic analysis result as a judgment basis in the vocabulary screening process, so that the dictionary updating no longer depends on the surface vocabulary information, but selects the vocabulary based on the semantic analysis result corresponding to the vocabulary, thereby accurately determining the target vocabulary suitable for the dictionary updating, improving the automation degree of the dictionary updating process and the accuracy of the vocabulary selection, and making the dictionary updating more reasonable and effective.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a dictionary updating method, apparatus, terminal device, and computer-readable storage medium. Background Technology

[0002] With the rapid growth of internet information, various types of online text data are constantly being generated, including a large number of newly emerging or semantically changed words. As a crucial foundational resource for language processing and information retrieval, dictionaries need continuous updates to maintain their timeliness and accuracy. Current dictionary update methods typically rely on simple keyword matching or basic text processing techniques, extracting words from massive amounts of data and then directly filtering or manually judging them. This makes it difficult to accurately identify the actual meaning and usage context of words, leading to the introduction of words without practical value or the omission of words with update value during the dictionary update process, thus affecting the accuracy and efficiency of dictionary updates. Summary of the Invention

[0003] This application provides a dictionary updating method, apparatus, terminal device, and computer-readable storage medium. It can introduce semantic analysis results as a judgment basis in the vocabulary selection process, so that the dictionary updating no longer depends solely on surface vocabulary information, but selects words based on the semantic analysis results corresponding to the words. This can more accurately determine the target words suitable for dictionary updating, improve the automation level of the dictionary updating process and the accuracy of vocabulary selection, and make the dictionary updating more reasonable and effective.

[0004] The technical solution adopted by this invention to solve the problem is as follows: On the one hand, this application provides a dictionary updating method, including: Obtain the vocabulary to be processed; Perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed; Based on the semantic analysis results, target words are identified from the words to be processed; Update the dictionary based on the target vocabulary.

[0005] In some embodiments of this application, semantic analysis is performed on the words to be processed to determine the semantic analysis results, including: Obtain device and scene information related to the words to be processed; Generate a semantic analysis request based on the words to be processed, device information, and scene information; The semantic analysis request is processed using natural language processing technology to obtain the semantic analysis results of the words to be processed.

[0006] In some embodiments of this application, the semantic analysis results include domain information, and target words are determined from the vocabulary to be processed based on the semantic analysis results, including: The domain information is compared with the domain whitelist corresponding to the dictionary. Among the words to be processed, words whose domain information belongs to the domain whitelist are identified as candidate words. Identify the target word from the candidate words.

[0007] In some embodiments of this application, identifying target words from candidate words includes: Based on the semantic analysis results of the candidate words, the candidate words are semantically judged to determine the chatty attribute value of the candidate words. The first value of the chatty attribute value indicates that the corresponding word is a chatty word, and the second value of the chatty attribute value indicates that the corresponding word is a non-chatty word. Candidate words with the second-highest value in the "casual conversation" attribute are identified as target words.

[0008] In some embodiments of this application, the semantic analysis results also include intent information and keyword information. Based on the semantic analysis results of the candidate words, semantic judgment is performed on the candidate words to determine the chatter attribute values ​​of the words to be processed, including: Fill the candidate words, their domain information, their intent information, and their keyword information into the preset prompt template to obtain the prompt information; The prompt information is input into the intelligent model, which performs semantic judgment on the candidate words and outputs the chatty attribute value of the candidate words.

[0009] In some embodiments of this application, the vocabulary to be processed is obtained, including: Obtain the data to be processed; Data cleaning is performed on the data to be processed to obtain the vocabulary to be processed. The data cleaning includes extracting the content of the first symbol, removing the content of the second symbol, removing the pre-defined punctuation marks, removing non-target language characters, removing version identifiers, removing duplicate words, removing subset identifiers, normalizing spaces, filtering single-character words and empty words, filtering pure numerical words, and filtering pure English names, or one or more of these steps.

[0010] In some embodiments of this application, before performing semantic analysis on the vocabulary to be processed and determining the semantic analysis result of the vocabulary to be processed, the following steps are also included: Acquire historical processed vocabulary data, as well as vocabulary from the dictionary; The words to be processed are compared with the words in the historical processed vocabulary data and the dictionary, respectively. Duplicate words are removed from the words to be processed. Duplicate words are those that exist in both the historical processed vocabulary data and the words to be processed, or those that exist in both the dictionary and the words to be processed.

[0011] Secondly, embodiments of the present invention also provide a dictionary updating device, comprising: The acquisition module is used to acquire the words to be processed. The analysis module is used to perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed. The determination module is used to determine the target word from the word to be processed based on the semantic analysis results of the word to be processed; The update module is used to update the dictionary based on the target vocabulary.

[0012] Thirdly, this application also provides a terminal device, which includes: One or more processors; Memory; and One or more applications, wherein the applications are stored in memory and configured to be executed by a processor to implement the dictionary update method of any of the first aspects.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the dictionary update method of any of the first aspects.

[0014] The beneficial effects of this invention are as follows: By acquiring the words to be processed and performing semantic analysis on them to determine the semantic analysis results, the target words are determined from the words to be processed based on the semantic analysis results, and then the dictionary is updated based on the target words. That is, the semantic analysis results are introduced as a judgment basis in the word selection process, so that the dictionary update no longer depends solely on the surface word information, but selects words based on the semantic analysis results corresponding to the words. This enables more accurate determination of target words suitable for dictionary updates, improves the automation level of the dictionary update process and the accuracy of word selection, and makes the dictionary update more reasonable and effective. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a dictionary update system provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating one embodiment of the dictionary update method provided in this invention. Figure 3 This is a flowchart illustrating a specific embodiment of the dictionary update method provided in this invention. Figure 4 This is a schematic block diagram of the dictionary update device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an embodiment of the terminal device provided in this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of the stated features.

[0019] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0020] It should be noted that since the method in this application embodiment is executed in a terminal device, the processing objects of each terminal device exist in the form of data or information, such as time, which is essentially time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the terminal device can process them. Specific details will not be elaborated here.

[0021] This application provides a dictionary updating method, apparatus, terminal device, and computer-readable storage medium, which will be described in detail below.

[0022] Please see Figure 1 , Figure 1 This is a schematic diagram of a dictionary update system provided in an embodiment of this application. The dictionary update system may include a terminal device 100, which integrates a dictionary update device, such as... Figure 1 Terminal devices in the process.

[0023] In this embodiment, the terminal device 100 is mainly used to acquire words to be processed; perform semantic analysis on the words to be processed to determine the semantic analysis results; determine target words from the words to be processed based on the semantic analysis results; and update the dictionary based on the target words. The semantic analysis results can be introduced as a judgment basis in the word selection process, so that the dictionary update no longer depends solely on surface word information, but selects words based on the semantic analysis results corresponding to the words. This enables more accurate determination of target words suitable for dictionary updates, improves the automation level of the dictionary update process and the accuracy of word selection, and makes the dictionary update more reasonable and effective.

[0024] In this embodiment, the terminal device 100 can be an independent server, a server network, or a server cluster. For example, the terminal device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0025] It is understood that the terminal device 100 used in the embodiments of this application can be a device that includes both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the terminal device 100 may be a desktop terminal or a mobile terminal, and the terminal device 100 may also be one of a mobile phone, tablet computer, laptop computer, etc.

[0026] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of more or fewer terminal devices shown, for example Figure 1 Only one terminal device is shown in the text. It is understood that the dictionary update system may also include one or more other services, which are not specified here.

[0027] In addition, such as Figure 1As shown, the dictionary update system may also include a memory 200 for storing data, such as vocabulary data, including words to be processed, candidate words, target words, etc.

[0028] It should be noted that, Figure 1 The schematic diagram of the dictionary update system shown is merely an example. The dictionary update system and scenario described in this application are intended to more clearly illustrate the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of dictionary update systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0029] First, this application provides a dictionary update method. The dictionary update method is executed by a dictionary update device, which is applied to a terminal device. The dictionary update method includes: acquiring words to be processed; performing semantic analysis on the words to be processed to determine the semantic analysis result; determining a target word in the words to be processed based on the semantic analysis result; and updating the dictionary based on the target word.

[0030] like Figure 2 The diagram shown is a flowchart of an embodiment of the dictionary update method in this application. The dictionary update method may include the following steps S201 to S203, as detailed below: Step S201: Obtain the vocabulary to be processed.

[0031] In this step, the vocabulary to be processed is the set of words obtained before the dictionary update process. The words in this set can originate from external text data or existing vocabulary data, and require further processing and analysis in subsequent steps. For example, when this application is applied to dictionary updates in the entertainment field, the vocabulary to be processed may include actor names, TV series names, movie names, variety show names, director names, etc., as well as nicknames used by viewers for actors or film / television productions, and other words that viewers might use when discussing entertainment information. It should be noted that not all of the above-mentioned words are suitable for inclusion in the dictionary. For example, some nicknames or temporary expressions are often arbitrary or non-standard and may not have long-term value as dictionary entries. Therefore, by obtaining these vocabulary to be processed, a relatively comprehensive vocabulary source can be formed first, providing a data foundation for subsequent analysis and screening, thereby ensuring that reasonable screening can be carried out during the dictionary update process while fully covering relevant vocabulary.

[0032] Step S202: Perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed.

[0033] In this step, semantic analysis is the process of analyzing and processing the meaning or semantic information expressed by words. It can identify or determine the semantic features or semantic attributes corresponding to words. The semantic analysis result represents the analysis result obtained after semantic analysis, which is used to characterize the semantic information of words.

[0034] Dictionary updates involve not only the words themselves, but also the meanings they express. If we only process words on their surface, we are prone to misjudgment. This step is used to determine the semantic information of words. Semantic analysis can identify the meaning of words in specific contexts, rather than just relying on the surface form of words. This provides a more reliable basis for subsequent judgments on whether words are worth adding to the dictionary.

[0035] In entertainment applications, some words may have multiple meanings in natural contexts. For example, some film and television drama titles or variety show names may also have ordinary word meanings in everyday language. Judging solely based on the surface form of a word may not be sufficient to determine whether it belongs to the entertainment field. By performing semantic analysis on the word to be processed, we can identify its specific meaning at the semantic level, such as whether it refers to a film or television work, variety show, or related person, thereby determining whether the word belongs to the expression in the entertainment field context.

[0036] Step S203: Based on the semantic analysis results, determine the target words from the words to be processed.

[0037] In this step, the target vocabulary refers to the words selected from the vocabulary to be processed for dictionary updates. In other words, the target vocabulary is the set of words that meet preset conditions and are chosen for dictionary updates. After obtaining the semantic analysis results of the words, the vocabulary can be judged and selected based on this semantic information. This allows the selection process to make decisions based on the meaning expressed by the words, determining words with update value as target vocabulary, rather than relying solely on the surface information of the words.

[0038] In entertainment applications, semantic analysis results can be used to retain more formal or stable words as target words, such as actor names, director names, film and television drama names, or variety show names. At the same time, expressions that only appear temporarily in specific contexts, lack stable meanings, or have obvious arbitrariness, such as nicknames or non-standard expressions spontaneously formed by some viewers, can be filtered out during the screening process.

[0039] Step S204: Update the dictionary based on the target vocabulary.

[0040] In this step, the identified target words can be added to the dictionary or the dictionary content can be adjusted accordingly, incorporating the selected target words into the dictionary system. Through this step, the dictionary can be continuously supplemented based on semantic analysis and screening results, thereby achieving dynamic updates to the dictionary content and improving the automation level of the dictionary update process and the accuracy of the update results.

[0041] By acquiring the words to be processed and performing semantic analysis on them to determine the results, and then identifying target words from the remaining words based on the results, the dictionary is updated accordingly. In other words, semantic analysis results are introduced as a criterion in the word selection process, so that the dictionary update no longer relies solely on surface-level word information, but selects words based on the semantic analysis results corresponding to the words. This allows for a more accurate identification of target words suitable for dictionary updates, improves the automation of the dictionary update process and the accuracy of word selection, and makes dictionary updates more reasonable and effective.

[0042] In one specific implementation, semantic analysis is performed on the words to be processed to determine the semantic analysis result of the words to be processed, including: obtaining device information and scene information related to the words to be processed; generating a semantic analysis request based on the words to be processed, device information and scene information; and processing the semantic analysis request based on natural language processing technology to obtain the semantic analysis result of the words to be processed.

[0043] This implementation constructs a semantic analysis request containing contextual information related to words, processes the request using Natural Language Processing (NLP), and introduces device information and scene information as auxiliary context, which can improve the accuracy of understanding the meaning of words and thus obtain more accurate semantic analysis results.

[0044] Device information refers to information related to the terminal device involved in the generation or use of the vocabulary, such as device type, device functional environment, or the system environment to which the device belongs. Context information refers to the application scenario or business environment in which the vocabulary is generated or used, such as specific business scenarios like entertainment, music, and games. A semantic analysis request is a standardized request data constructed according to a preset format, used to submit the vocabulary to be analyzed and its contextual information to the semantic understanding module or natural language processing module. Natural language processing technology is a computer processing technology used to understand and analyze human language. By analyzing input text, it can identify the semantic information expressed in the text. In this scheme, the semantic analysis results can include multiple aspects of semantic information, such as domain information, intent information, and keyword information (or slots). Specifically, "domain" refers to the business field to which the word belongs, such as the fields of movies, music, or games in an entertainment-related context; "intent" refers to the behavior or need that may be expressed when using the word, such as searching for a film or television work, learning about actors, or watching a program; and "slots" refer to the key semantic information extracted from the word, which is used to describe the specific object or meaning that the word refers to.

[0045] The meaning of words is often closely related to their usage environment; the same word may have different semantic connotations in different devices or business scenarios. In this implementation, the device and scenario information related to the word to be processed are first obtained. This provides additional contextual conditions for semantic understanding, allowing subsequent semantic analysis to be conducted in a more explicit context. For example, in an entertainment application scenario, when the word to be processed is a movie title, if the word comes from a smart TV or video playback device and is in an entertainment content browsing scenario, it is more likely to represent the movie title than a general word meaning. Secondly, a semantic analysis request is generated based on the word to be processed, device information, and scenario information. This request is then processed using natural language processing (NLP) technology to obtain the semantic analysis result of the word to be processed. This allows NLP technology to simultaneously refer to the word content and its contextual information when processing words, thereby improving the accuracy of semantic understanding and recognizing the semantic information expressed in the text. For example, in entertainment applications, when the word to be processed is the title of a film or television series, natural language processing technology can identify that the word belongs to the film or television series field, extract key information such as the corresponding work title, and determine that the user's intention in using the word is to refer to a specific film or television series, thus forming a complete semantic analysis result.

[0046] In one specific implementation, the semantic analysis result includes domain information. Based on the semantic analysis result, the target word is determined from the vocabulary to be processed, including: comparing the domain information with the domain whitelist corresponding to the dictionary; determining the words in the domain information that belong to the domain whitelist as candidate words from the vocabulary to be processed; and determining the target word from the candidate words.

[0047] This implementation method utilizes domain information from semantic analysis results to initially screen the vocabulary to be processed, thereby determining candidate words. Based on these candidate words, it further determines the target words for updating the dictionary. The semantic analysis results include domain information, which is a semantic label representing the business domain to which a word belongs. For example, natural language processing technology can identify that a word is more likely to be related to business domains such as movies, music, games, or everyday conversations, thus assigning the word corresponding domain information for classifying words based on business domains and identifying the application scope of words from a semantic perspective. A domain whitelist refers to a pre-defined set of domains allowed to be included in the processing flow for a specific type of dictionary. Each dictionary can be configured with a corresponding domain whitelist to limit the business scope that the dictionary focuses on. For example, when the dictionary is a film and television dictionary in the entertainment field, its domain whitelist may include tags such as "movie" and "chitchat." "Movie" can represent the field related to film and television works, and "chitchat" can represent everyday conversational expressions related to entertainment topics. Domain whitelists can pre-define the acceptable domain range for the corresponding dictionary, quickly filtering out words that clearly do not belong to the target domain, thereby reducing the interference of irrelevant words on subsequent processing. Candidate words are the set of words that are retained after domain-based filtering from the words to be processed, and are qualified to enter the subsequent further judgment process.

[0048] In this implementation, domain information is first determined based on the semantic analysis results. This domain information is then compared with the corresponding domain whitelist in the dictionary. If the domain information obtained from semantic analysis does not belong to the domain whitelist, it indicates that the word may not semantically fall within the domain scope of the current dictionary. For example, in the scenario of updating an entertainment dictionary, if a word to be processed is identified as belonging to another business domain after semantic analysis, then this word is not suitable for direct participation in the subsequent automatic filtering process. In this case, the word can be marked as requiring manual review, and its corresponding semantic analysis information, such as domain information, intent information, and slot information, can be recorded for further manual judgment. If the domain information obtained from semantic analysis belongs to the domain whitelist, it indicates that the word may semantically belong to the business domain of the current dictionary. In this case, the word can be identified as a candidate word, and then the target word can be further determined from the candidate words.

[0049] In one specific implementation, determining the target word from the candidate words includes: performing semantic judgment on the candidate words based on the semantic analysis results of the candidate words, determining the chatty attribute value of the candidate words, wherein a first value of the chatty attribute value indicates that the corresponding word is a chatty word, and a second value of the chatty attribute value indicates that the corresponding word is a non-chatty word; and determining the candidate words with the second value of the chatty attribute value as the target words.

[0050] This implementation introduces a chatty attribute judgment mechanism based on semantic understanding into the candidate words obtained through domain screening. This mechanism further distinguishes between words with dictionary inclusion value and expressions used only in daily communication, thereby determining the target words for updating the dictionary. Semantic judgment is a process of further identifying the semantic category of words based on existing semantic analysis information. Chatty attributes are semantic attributes used to characterize whether a word belongs to everyday conversation or casual expression. For example, in online discussions or user interactions, words used to express emotions, evaluations, or jokes often appear. While these words may appear in entertainment-related contexts, they are not necessarily suitable as standard entries in the dictionary. By judging the chatty attributes of candidate words, chatty expressions can be identified, thus distinguishing them during dictionary updates. The chatty attribute value is the numerical expression of the chatty attribute judgment result. A first value indicates that the corresponding word is a chatty word, while a second value indicates that the corresponding word is not a chatty word.

[0051] Although candidate words have been filtered through a domain whitelist, they may still contain a large number of words used only for communication or commentary, such as emotional expressions, teasing terms, or temporary phrases. These words usually lack stable meanings or long-term usage value, thus requiring further differentiation through more detailed semantic judgment. In this implementation, based on existing semantic analysis information, the candidate words can be judged for their chattiness attributes. Chattiness expressions unsuitable for inclusion as dictionary entries can be identified, marked as requiring manual review, and the candidate words judged as non-chatiness words can be used as the final words for updating the dictionary. The judgment result for chattiness attributes can be represented using Boolean values; a true value indicates that the word belongs to the chattiness category, while a false value indicates that the word does not belong to the chattiness category.

[0052] In one specific implementation, the semantic analysis results also include intent information and keyword information. Based on the semantic analysis results of the candidate words, semantic judgment is performed on the candidate words to determine the chatter attribute value of the words to be processed. This includes: filling the candidate words, the domain information of the candidate words, the intent information of the candidate words, and the keyword information of the candidate words into a preset prompt template to obtain prompt information; inputting the prompt information into an intelligent model, which performs semantic judgment on the candidate words and outputs the chatter attribute value of the candidate words.

[0053] This implementation utilizes semantic analysis results obtained through natural language processing (NLP) technology to construct prompts containing various semantic information. These prompts are then input into an intelligent model for semantic judgment, thereby determining whether candidate words belong to the category of casual conversation. The semantic analysis results include domain information, intent information, and keyword information. Intent information refers to the behavior or need that a word might express when used, identified through NLP technology. For example, a user using a word might want to search for, obtain information about, or discuss a particular film or television series. Keyword information is extracted from words to characterize their specific meaning, describing the object or core semantics they refer to. The prompt template is a pre-defined structured text template used to organize the semantic information to be provided to the intelligent model. The prompt information is the complete input content formed by filling specific words and their semantic information into this template, guiding the intelligent model to perform a specific semantic judgment task. The intelligent model can refer to a model with semantic understanding capabilities, such as a Large Language Model (LLM), which can comprehensively analyze the semantics of words based on the input prompts and provide a judgment result.

[0054] In this implementation, candidate words, their domain attributes, intent information, and keyword information are filled into a preset prompt template to obtain prompt information. This prompt information is then input into a large language model, which performs semantic judgment on the candidate words and outputs their casual conversation attribute values. When performing semantic understanding, the judgment results of the large language model are often closely related to the completeness of the input information and the richness of the context. If only a single word is provided as input, the model may struggle to accurately determine the semantic attributes of that word in a specific scenario. By adding domain attributes, intent information, and keyword information to the prompt information, a more complete semantic background can be provided to the model, thereby improving its judgment ability.

[0055] The above approach utilizes the semantic analysis results output by NLP as the input basis for LLM judgments. This allows LLM to simultaneously refer to the words themselves and their semantic analysis results when judging chatter attributes, thereby improving the accuracy of semantic judgments. This not only reduces the errors caused by judging solely based on the surface information of words, but also fully leverages the semantic understanding capabilities of intelligent models, making the word selection process more intelligent and automated. This further improves the accuracy of target word determination and overall processing efficiency during dictionary updates.

[0056] In one specific implementation, obtaining the vocabulary to be processed includes: obtaining data to be processed; performing data cleaning on the data to be processed to obtain the vocabulary to be processed, wherein the data cleaning includes extracting the content of the first symbol, removing the content of the second symbol, removing pre-set punctuation marks, removing non-target language characters, removing version identifiers, removing duplicate words, removing subset identifiers, space normalization, filtering single-character words and empty words, filtering pure numeric words, and filtering pure English names, or one or more of these.

[0057] This implementation introduces a data cleaning mechanism, which can effectively improve the standardization and accuracy of vocabulary data, thereby providing a reliable data foundation for subsequent semantic analysis and vocabulary selection. The data to be processed is raw data collected from external data sources, which can exist in the form of text or structured information, containing a large amount of names, tags, or descriptive information related to entertainment content. The first symbol can represent symbols used to identify the main content of the vocabulary, such as book title marks 《》. In the entertainment field, film and television titles are usually enclosed in book title marks. The second symbol can represent symbols used to annotate explanatory information, such as parentheses, which often contain explanatory content, such as actor introductions or additional explanations.

[0058] Specifically, vocabulary-related information can be obtained from multiple heterogeneous data sources through the data acquisition module. Since the data formats and interface methods of different data sources may vary, a unified access method can be used for various data sources. This involves configuring parameters such as interface addresses, data formats, and parsing rules to achieve unified management and automatic acquisition of multiple data sources. In the actual data acquisition process, to improve system processing efficiency, an asynchronous callback processing mechanism can be used. After sending a data request to the dictionary service, the system continuously monitors the generation of callback data files through an independent callback listening thread, and only parses the data after it is detected that the writing is complete. To avoid reading data files that have not yet been fully written, a file integrity verification mechanism can be used. For example, the file size is recorded first, and then checked again after a certain time interval. If the file size remains unchanged, the data writing is considered complete. In addition, a timeout protection mechanism can be set to automatically terminate the waiting and log the process when a preset waiting time is exceeded, thus preventing prolonged system blocking. These mechanisms improve the stability and concurrent processing capabilities during large-scale data acquisition.

[0059] After acquiring the data to be processed, data cleaning is performed to obtain the vocabulary to be processed. This involves standardizing the raw data through a series of preset rules to remove irrelevant information, noisy data, or content that does not meet the requirements, making the final vocabulary data more accurate and consistent. The data cleaning process can include various operations, such as extracting content from book titles (first symbol) as the main vocabulary to obtain standardized film and television titles; removing content from parentheses (second symbol) to avoid interference from annotation information in vocabulary recognition; removing pre-defined punctuation marks, i.e., removing special punctuation marks from the raw data, such as hyphens, commas, square brackets, quotation marks, asterisks, or hash symbols; removing non-target language characters, i.e., using character category recognition methods to retain only characters that meet the requirements of the target language, such as Chinese, Latin letters, numbers, and common punctuation, while filtering out non-target language content such as Japanese kana, Korean characters, Russian characters, Arabic characters, or emoticons, thus ensuring the linguistic consistency of the vocabulary data; and removing various version identifiers, such as removing "..." Descriptive labels such as "Full Version," "3D Version," "IMAX Version," or "2023 Version" are primarily used to differentiate between different releases, but they do not have core semantic value for basic dictionary entries. It can also remove identifiers indicating subset relationships, such as "Season x," "Episode x," or "Part x." Furthermore, it can perform space normalization, uniformly adjusting spaces in words, removing spaces between non-English words, merging multiple consecutive spaces into a single space, and removing redundant spaces at the beginning and end of words. It can also filter single-character words and empty words, filter purely numerical words, and perform special filtering based on specific dictionary types. For example, in an actor dictionary scenario, since actors usually have Chinese translations, purely English names can be filtered to prevent invalid words from entering the dictionary. In addition, it can use set structures to deduplicate words, automatically removing duplicates and improving subsequent processing efficiency.

[0060] In one specific implementation, before performing semantic analysis on the words to be processed and determining the semantic analysis result of the words to be processed, the method further includes: acquiring historical processed word data and words in a dictionary; comparing the words to be processed with the historical processed word data and words in the dictionary respectively, and removing duplicate words from the words to be processed, wherein duplicate words are words that exist in both the historical processed word data and the words to be processed, or words that exist in both the dictionary and the words to be processed.

[0061] In this embodiment, the vocabulary in the dictionary refers to the existing entries in the current dictionary, that is, the set of vocabulary that has been confirmed and stored in the dictionary. Historical processing vocabulary data consists of records of vocabulary that have been processed in previous processing flows.

[0062] Dictionary updates typically involve periodically acquiring new vocabulary data from multiple data sources. These data sources may have significant overlap across different time periods. Without historical comparison, the same words might be repeatedly subjected to semantic analysis and filtering, resulting in unnecessary computational overhead. Therefore, the word to be processed can be compared with both historically processed vocabulary data and words in the dictionary to determine if they are identical. This helps identify duplicate words. Duplicate words fall into two categories: those present in both historically processed vocabulary data and the word to be processed (indicating they have already been processed in previous steps), and those present in both the dictionary and the word to be processed (indicating they have already been included as entries in the dictionary). After identifying duplicate words, they can be removed from the word to be processed, leaving only unprocessed words not yet included in the dictionary.

[0063] As one specific embodiment, such as Figure 3As shown, the process begins by acquiring business-related text data from external data sources as raw data, such as film and television information, actor information, or related tag information in the entertainment field. Next, the raw data is cleaned by extracting standardized vocabulary and removing irrelevant or noisy data to obtain the data to be processed. For example, in an entertainment application scenario, film and television titles within quotation marks can be extracted from the text, while removing explanatory information, special symbols, or version identifiers within parentheses, and filtering for spaces, duplicate words, and non-compliant characters. After obtaining the vocabulary to be processed, NLP technology is used to determine the domain to which the vocabulary belongs. Taking the entertainment field as an example, it can be identified whether a word belongs to the fields of film and television, music, games, or everyday conversation. Subsequently, it is determined whether the domain is within the dictionary's domain whitelist. For example, in a film and television dictionary application scenario, the whitelist may include film and television-related domain tags. If the domain attribute of a word does not fall within the whitelist, it indicates that the word may not belong to the business domain currently being studied by the dictionary. In this case, the reason for the word failing the screening is recorded, and it is submitted for manual review. If a word's domain attribute falls within the domain whitelist, it indicates that the word may semantically belong to the domain the dictionary focuses on. In this case, the next processing step involves constructing a prompt message to input into the large language model for further semantic judgment. This determines whether the word belongs to casual conversation, such as a teasing or emotional expression used by users when discussing entertainment topics. After obtaining the judgment result, it is determined whether the result indicates the word is not casual. If the word is determined to be casual, it indicates that the word may lack stable semantics or standardization. In this case, the relevant information is recorded and submitted for manual review to prevent inappropriate words from being automatically added to the dictionary. If the word is determined to be non-casual, it indicates that the word has a relatively clear and stable semantic expression. In this case, the word is added to the dictionary as a new entry according to the preset dictionary format, thus completing the dictionary content update.

[0064] To better implement the dictionary update method in the embodiments of this application, a dictionary update device is also provided in the embodiments of this application, such as... Figure 4 As shown, the dictionary update device 400 includes: Module 410 is used to acquire words to be processed; Analysis module 420 is used to perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed. The determination module 430 is used to determine the target word in the vocabulary to be processed based on the semantic analysis results of the vocabulary to be processed; Update module 440 is used to update the dictionary based on the target vocabulary.

[0065] In this embodiment, the process involves acquiring words to be processed and performing semantic analysis on them to determine the results. Based on the semantic analysis results, target words are identified from the words to be processed, and the dictionary is then updated based on the target words. In other words, semantic analysis results are introduced as a criterion during the word selection process, so that the dictionary update no longer relies solely on surface-level word information, but rather selects words based on the semantic analysis results corresponding to the words. This allows for a more accurate determination of target words suitable for dictionary updates, improving the automation level of the dictionary update process and the accuracy of word selection, making the dictionary update more reasonable and effective.

[0066] In some embodiments of this application, the analysis module 420 performs semantic analysis on the words to be processed and determines the semantic analysis results of the words to be processed, including: Obtain device and scene information related to the words to be processed; Generate a semantic analysis request based on the words to be processed, device information, and scene information; The semantic analysis request is processed using natural language processing technology to obtain the semantic analysis results of the words to be processed.

[0067] In some embodiments of this application, the semantic analysis results include domain information. Based on the semantic analysis results, the determining module 430 determines the target vocabulary in the vocabulary to be processed, including: The domain information is compared with the domain whitelist corresponding to the dictionary. Among the words to be processed, words whose domain information belongs to the domain whitelist are identified as candidate words. Identify the target word from the candidate words.

[0068] In some embodiments of this application, the determining module 430 determines the target word from the candidate words by including: Based on the semantic analysis results of the candidate words, the candidate words are semantically judged to determine the chatty attribute value of the candidate words. The first value of the chatty attribute value indicates that the corresponding word is a chatty word, and the second value of the chatty attribute value indicates that the corresponding word is a non-chatty word. Candidate words with the second-highest value in the "casual conversation" attribute are identified as target words.

[0069] In some embodiments of this application, the semantic analysis results also include intent information and keyword information. The determining module 430 performs semantic judgment on the candidate words based on the semantic analysis results of the candidate words to determine the chatter attribute value of the words to be processed, including: Fill the candidate words, their domain information, their intent information, and their keyword information into the preset prompt template to obtain the prompt information; The prompt information is input into the intelligent model, which performs semantic judgment on the candidate words and outputs the chatty attribute value of the candidate words.

[0070] In some embodiments of this application, the acquisition module 410 acquires the words to be processed, including: Obtain the data to be processed; Data cleaning is performed on the data to be processed to obtain the vocabulary to be processed. The data cleaning includes extracting the content of the first symbol, removing the content of the second symbol, removing the pre-defined punctuation marks, removing non-target language characters, removing version identifiers, removing duplicate words, removing subset identifiers, normalizing spaces, filtering single-character words and empty words, filtering pure numerical words, and filtering pure English names, or one or more of these steps.

[0071] In some embodiments of this application, before performing semantic analysis on the words to be processed and determining the semantic analysis result of the words to be processed, the acquisition module 410 is further configured to: Acquire historical processed vocabulary data, as well as vocabulary from the dictionary; The words to be processed are compared with the words in the historical processed vocabulary data and the dictionary, respectively. Duplicate words are removed from the words to be processed. Duplicate words are those that exist in both the historical processed vocabulary data and the words to be processed, or those that exist in both the dictionary and the words to be processed. This application embodiment also provides a terminal device that integrates any of the dictionary updating devices provided in this application embodiment. The terminal device includes: One or more processors; Memory; and One or more applications, wherein the applications are stored in memory and configured to be executed by a processor from the steps of the dictionary update method in any of the embodiments described above.

[0072] This application also provides a terminal device that integrates any of the dictionary update devices provided in this application. For example... Figure 5 As shown, it illustrates a structural schematic diagram of the terminal device involved in the embodiments of this application. Specifically: The terminal device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 501 is the control center of the terminal device. It connects various parts of the terminal device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 502, and by calling data stored in the memory 502, thereby providing overall monitoring of the terminal device. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0073] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0074] The terminal device also includes a power supply 503 that supplies power to the various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0075] The terminal device may also include an input unit 504, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0076] Although not shown, the terminal device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the terminal device loads the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 runs the applications stored in the memory 502 to realize various functions, as follows: Obtain the vocabulary to be processed; Perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed; Based on the semantic analysis results, target words are identified from the words to be processed; Update the dictionary based on the target vocabulary.

[0077] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0078] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the dictionary update methods provided in embodiments of this application. For example, the computer program loaded by the processor can execute the following steps: Obtain the vocabulary to be processed; Perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed; Based on the semantic analysis results, target words are identified from the words to be processed; Update the dictionary based on the target vocabulary.

[0079] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0080] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0081] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0082] The dictionary updating method, apparatus, terminal device, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A dictionary updating method, characterized in that, include: Obtain the vocabulary to be processed; Perform semantic analysis on the words to be processed, and determine the semantic analysis results of the words to be processed; Based on the semantic analysis results, target words are determined from the words to be processed. The dictionary is updated based on the target vocabulary.

2. The dictionary updating method according to claim 1, characterized in that, The step of performing semantic analysis on the words to be processed and determining the semantic analysis results of the words to be processed includes: Obtain device and scene information related to the vocabulary to be processed; Generate a semantic analysis request based on the vocabulary to be processed, the device information, and the scene information; The semantic analysis request is processed using natural language processing technology to obtain the semantic analysis results of the words to be processed.

3. The dictionary updating method according to claim 1, characterized in that, The semantic analysis results include domain information, and the step of determining target words from the vocabulary to be processed based on the semantic analysis results includes: The domain information is compared with the domain whitelist corresponding to the dictionary, and among the words to be processed, words whose domain information belongs to the domain whitelist are determined as candidate words; The target word is determined from the candidate words.

4. The dictionary updating method according to claim 3, characterized in that, The step of determining the target word from the candidate words includes: Based on the semantic analysis results of the candidate words, the candidate words are semantically judged to determine the chatty attribute value of the candidate words. The chatty attribute value is a first value indicating that the corresponding word is a chatty word, and the chatty attribute value is a second value indicating that the corresponding word is a non-chatty word. Candidate words with a chat attribute value of the second value are identified as target words.

5. The dictionary updating method according to claim 4, characterized in that, The semantic analysis results also include intent information and keyword information. The semantic judgment of the candidate words based on the semantic analysis results, determining the casual conversation attribute value of the words to be processed, includes: The candidate words, their domain information, their intent information, and their keyword information are filled into a preset prompt template to obtain prompt information. The prompt information is input into the intelligent model, which performs semantic judgment on the candidate words and outputs the chatty attribute value of the candidate words.

6. The dictionary updating method according to claim 1, characterized in that, The process of obtaining the words to be processed includes: Obtain the data to be processed; The data to be processed is cleaned to obtain the vocabulary to be processed. The data cleaning includes extracting the content of the first symbol, removing the content of the second symbol, removing the pre-set punctuation marks, removing non-target language characters, removing version identifiers, removing duplicate words, removing subset identifiers, normalizing spaces, filtering single-character words and empty words, filtering pure numerical words, and filtering pure English names, or one or more of these.

7. The dictionary updating method according to claim 1, characterized in that, Before performing semantic analysis on the words to be processed and determining the semantic analysis result of the words to be processed, the method further includes: Acquire historical processed vocabulary data, as well as the vocabulary in the dictionary; The words to be processed are compared with the words in the historical processed word data and the words in the dictionary, respectively. Duplicate words are removed from the words to be processed. The duplicate words are words that exist in both the historical processed word data and the words to be processed, or words that exist in both the dictionary and the words to be processed.

8. A dictionary updating device, characterized in that, include: The acquisition module is used to acquire the words to be processed. The analysis module is used to perform semantic analysis on the words to be processed and determine the semantic analysis results of the words to be processed. The determination module is used to determine the target word from the word to be processed based on the semantic analysis results of the word to be processed; The update module is used to update the dictionary based on the target vocabulary.

9. A terminal device, characterized in that, The terminal device includes: one or more processors, a memory, and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the dictionary update method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the dictionary update method according to any one of claims 1 to 7.

Images