Terminology construction method and device, electronic equipment, storage medium and computer program product

Abstract

The present disclosure relates to a vocabulary construction method and device, electronic equipment, storage medium and computer program product, comprising: performing word segmentation processing on a target text to obtain a plurality of words; for each word in the plurality of words, comparing the word with a plurality of preset anchor keywords, and in the case where the word matches a target preset anchor keyword in the plurality of preset anchor keywords, assigning the word a category corresponding to the target preset anchor keyword; inputting each word in the plurality of words and the category assigned to each word into a large language model; correcting the category assigned to each word by the large language model to obtain a vocabulary. The present disclosure can provide an automatic vocabulary construction method, which can effectively reduce the workload of manually maintaining the vocabulary and repeatedly supplementing words, thereby reducing the maintenance cost. Moreover, the present disclosure does not require a large amount of high-quality annotated corpus as a prerequisite, reduces the dependence on prior dictionaries and annotated data, and can be quickly implemented.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to methods, apparatus, electronic devices, storage media, and computer program products for constructing a thesaurus. Background Technology

[0002] Texts describing equipment quality issues typically originate from duty logs, inspection records, maintenance logs, and oral transcriptions, exhibiting clear characteristics of colloquialism, abbreviation, and contextualization. For example, descriptions of the same object may simultaneously appear in full, abbreviation, colloquial, and misspelled forms. Therefore, raw texts are often difficult to use directly as domain vocabularies and require processing to become stable and usable domain vocabularies.

[0003] In related technologies, there are various ways to convert raw text into a domain-specific vocabulary, such as relying on manually constructed industry dictionaries, or relying on supervised named entity recognition or relation extraction models. However, each of these methods has its own limitations. Specifically, while relying on manually constructed industry dictionaries is direct to implement, its coverage is limited when dealing with real-world equipment quality issues, and its maintenance costs increase rapidly as the scenario expands. Supervised learning methods, on the other hand, are heavily reliant on high-quality labeled corpora, making rapid deployment difficult. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for constructing a dictionary, in order to at least solve the problems of high maintenance costs associated with manually constructing industry dictionaries and the difficulty in rapidly implementing supervised learning methods in the aforementioned related technologies.

[0005] According to a first aspect of the present disclosure, a vocabulary construction method is provided, comprising: acquiring target text; performing word segmentation on the target text to obtain multiple words; comparing each of the multiple words with multiple preset anchor keywords, and assigning a category corresponding to the target preset anchor keyword to the word if the word matches a target preset anchor keyword among the multiple preset anchor keywords; inputting each of the multiple words and the category assigned to each word into a large language model; and correcting the category assigned to each word through the large language model to obtain a vocabulary, wherein the vocabulary includes at least each of the multiple words and the corrected category corresponding to each word.

[0006] Optionally, before performing word segmentation on the target text to obtain multiple words, the method further includes: preprocessing the target text to obtain preprocessed text, wherein the preprocessing includes at least one of the following: removing punctuation marks contained in the target text, deleting spaces contained in the target text; the word segmentation on the target text to obtain multiple words includes: performing word segmentation on the preprocessed text to obtain the multiple words.

[0007] Optionally, the step of segmenting the target text to obtain multiple words includes: segmenting the target text using at least two segmentation methods to obtain the multiple words.

[0008] Optionally, before comparing each of the plurality of words with a plurality of preset anchor keywords, the method further includes: filtering the plurality of words using at least one filtering rule to obtain words retained after filtering, wherein the at least one filtering rule includes at least one of the following: the frequency of the word is greater than or equal to a preset frequency threshold, the length of the word meets a preset length requirement, the word is not a preset stop word, the word is not a pure string of numbers, the word is not a pure string of symbols, and the number of words retained after filtering is less than or equal to a preset number threshold; the comparison of each of the plurality of words with a plurality of preset anchor keywords includes: comparing each of the words retained after filtering with the plurality of preset anchor keywords.

[0009] Optionally, for each of the plurality of words, comparing the word with a plurality of preset anchor keywords, and assigning the word to a category corresponding to the target preset anchor keyword if the word matches a target preset anchor keyword among the plurality of preset anchor keywords, includes: For each of the plurality of words, if the word contains any one of the target preset anchor keywords under the target category, the target category corresponding to the target preset anchor keyword is assigned to the word. Each of the plurality of preset categories corresponds to at least one preset anchor keyword among the plurality of preset anchor keywords, and the target category is a preset category among the plurality of preset categories.

[0010] Optionally, it further includes: obtaining and outputting the mapping relationship between non-standard expressions and standard expressions for the same word through the large language model, wherein the non-standard expressions include at least one of the following: colloquial expressions, abbreviations, slang expressions, and incorrect expressions, and both the non-standard expressions and the standard expressions originate from the plurality of words.

[0011] Optionally, before correcting the categories assigned to each word using the large language model to obtain a vocabulary, the method further includes: using the large language model to remove noise from multiple words input to the large language model to obtain words retained after noise removal, wherein the noise removal includes at least one of the following: removing meaningless colloquial words, removing general words unrelated to the business theme expressed by the target text; the step of correcting the categories assigned to each word using the large language model to obtain a vocabulary includes: using the large language model to correct the categories assigned to the words retained after noise removal to obtain the vocabulary.

[0012] Optionally, after correcting the categories assigned to each word using the large language model to obtain a vocabulary, the method further includes: for each word in the vocabulary, determining, using the large language model, whether the category corresponding to the word is within a preset category whitelist, and / or whether the word originates from the multiple words; if the category corresponding to the word is within the preset category whitelist, and / or the word originates from the multiple words, maintaining the word and its corresponding category within the vocabulary; otherwise, deleting the word and its corresponding category from the vocabulary.

[0013] According to a second aspect of the present disclosure, a vocabulary construction apparatus is provided, comprising: a text acquisition module configured to acquire target text; a word segmentation module configured to segment the target text to obtain multiple words; a category assignment module configured to compare each of the multiple words with multiple preset anchor keywords, and assign a category corresponding to the target preset anchor keyword to the word if the word matches a target preset anchor keyword among the multiple preset anchor keywords; an input module configured to input each of the multiple words and the category assigned to each word into a large language model; and a category correction module configured to correct the category assigned to each word through the large language model to obtain a vocabulary, wherein the vocabulary includes at least each of the multiple words and the corrected category corresponding to each word.

[0014] Optionally, the vocabulary construction device further includes: a preprocessing module configured to preprocess the target text to obtain preprocessed text, wherein the preprocessing includes at least one of the following: removing punctuation marks contained in the target text, deleting spaces contained in the target text; and the word segmentation module configured to perform word segmentation processing on the preprocessed text to obtain the plurality of words.

[0015] Optionally, the word segmentation module is configured to perform word segmentation on the target text using at least two word segmentation methods to obtain the multiple words.

[0016] Optionally, the vocabulary construction device further includes: a filtering module configured to filter the plurality of words using at least one filtering rule to obtain words retained after filtering, wherein the at least one filtering rule includes at least one of the following: the frequency of the word is greater than or equal to a preset frequency threshold, the length of the word meets a preset length requirement, the word is not a preset stop word, the word is not a pure number string, the word is not a pure symbol string, and the number of words retained after filtering is less than or equal to a preset number threshold; the category allocation module is configured to: for each word retained after filtering, compare the word with the plurality of preset anchor keywords.

[0017] Optionally, the category allocation module is configured to: for each of the plurality of words, if the word contains any one of the target preset anchor keywords under the target category, assign the target category corresponding to the target preset anchor keyword to the word, wherein each of the plurality of preset categories corresponds to at least one preset anchor keyword among the plurality of preset anchor keywords, and the target category is a preset category among the plurality of preset categories.

[0018] Optionally, the vocabulary construction device further includes a mapping relationship acquisition module, configured to acquire and output the mapping relationship between non-standard expressions and standard expressions for the same word through the large language model, wherein the non-standard expressions include at least one of the following: colloquial expressions, abbreviations, slang expressions, and incorrect expressions, and both the non-standard expressions and the standard expressions originate from the plurality of words.

[0019] Optionally, the vocabulary construction device further includes: a noise removal module, configured to remove noise from multiple words input to the large language model using the large language model to obtain words retained after noise removal, wherein the noise removal includes at least one of the following: removing meaningless colloquial words, removing general words unrelated to the business theme expressed by the target text; the category correction module is configured to: correct the categories assigned to the words retained after noise removal using the large language model to obtain the vocabulary.

[0020] Optionally, the vocabulary construction device further includes: a judgment module, configured to, for each word contained in the vocabulary, determine, through the large language model, whether the category corresponding to the word is within a preset category whitelist, and / or whether the word originates from the multiple words; a maintenance module, configured to, if the category corresponding to the word is within the preset category whitelist, and / or the word originates from the multiple words, maintain the word and its corresponding category within the vocabulary; and a deletion module, configured to, otherwise, delete the word and its corresponding category from the vocabulary.

[0021] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a vocabulary construction method according to the present disclosure.

[0022] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a vocabulary construction method according to the present disclosure.

[0023] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a vocabulary construction method according to the present disclosure.

[0024] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In this disclosure, compared to the reliance on manually constructed industry dictionaries in related technologies, an automated vocabulary construction method can be provided, effectively reducing the workload of manual vocabulary maintenance and repeated word addition, thereby lowering maintenance costs. Furthermore, compared to the reliance on supervised named entity recognition or relation extraction models in related technologies for vocabulary construction, this disclosure does not require a large-scale, high-quality labeled corpus; instead, vocabulary construction can be initiated using only the original equipment quality problem text. This reduces the dependence on prior dictionaries and labeled data, enabling rapid deployment and making it more suitable for cold starts and small-sample scenarios involving equipment quality data.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 This is a flowchart illustrating a vocabulary construction method according to an exemplary embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a specific implementation of a vocabulary construction method according to exemplary embodiments of the present disclosure; Figure 3 This is a block diagram illustrating a vocabulary construction apparatus according to exemplary embodiments of the present disclosure; Figure 4 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0028] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0029] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0030] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0031] Figure 1 This is a flowchart illustrating a vocabulary construction method according to an exemplary embodiment of the present disclosure.

[0032] Reference Figure 1 In step 101, the target text can be obtained. For example, the original equipment quality problem text can be obtained, which usually comes from duty records, inspection records, maintenance logs, and oral transcriptions, etc.

[0033] In step 102, the target text can be segmented to obtain multiple words. For example, words or phrases with practical significance in the field of equipment quality can be automatically extracted from the aforementioned original text on equipment quality issues.

[0034] According to an exemplary embodiment of this disclosure, the target text can be preprocessed to obtain preprocessed text. This preprocessing may include at least one of the following: removing punctuation marks from the target text and deleting spaces from the target text. Then, the preprocessed text can be segmented to obtain multiple words. That is, in this disclosure, before segmenting the target text, preprocessing such as symbol cleaning and whitespace normalization can be performed to make the text format more standardized and normalized, thus laying a foundation for the subsequent segmentation stage.

[0035] According to exemplary embodiments of this disclosure, target text can be segmented using at least two segmentation methods to obtain multiple words. For example, words can be extracted from the target text using the jieba segmentation channel and the regular expression supplementary channel from related technologies. The jieba segmentation channel is primarily used to extract regular words and phrases from the text; the regular expression supplementary channel is primarily used to capture model numbers, abbreviations, equipment numbers, component abbreviations, and common phrase expressions found in field records from the text.

[0036] It should be noted that the word frequency of each word extracted from the target text can also be calculated using the following dual-channel method. This word frequency specifically refers to the number of times the same word appears:

[0037] in, This represents the total word frequency of word t. This represents the word frequency of word t as calculated using the jieba word segmentation channel. This represents the word frequency of word t obtained through supplementary channel statistics using regular expressions.

[0038] Thus, by employing a word extraction mechanism that combines the jieba segmentation channel with a regular expression supplementation channel, this disclosure can obtain a high-recall word set from colloquial and abbreviated equipment quality issue texts without relying on mature professional dictionaries. Furthermore, the dual-channel candidate word discovery mechanism provided in this disclosure can effectively improve the coverage of colloquial expressions, abbreviations, slang, and erroneous descriptions. Compared to methods that rely entirely on manual vocabularies and rule templates, it can effectively reduce the workload of manual item-by-item maintenance and repeated word supplementation.

[0039] In step 103, for each of the multiple words, the word can be compared with multiple preset anchor keywords. If the word matches a target preset anchor keyword among the multiple preset anchor keywords, the word can be assigned a category corresponding to the target preset anchor keyword. Here, preset anchor keywords can refer to a set of prior keywords used for heuristically determining the category of a word. For example, in the field of device quality, the category of a word can include, but is not limited to: equipment, component, environment, phenomenon, action, result, cause, parameter, etc.

[0040] Thus, in this disclosure, by using preset anchor keywords to construct a heuristic coarse classification mechanism, words can be mapped to the corresponding categories under unlabeled conditions, thereby providing prior constraints for subsequent refined bootstrapping.

[0041] Furthermore, if a word cannot be successfully matched with any preset anchor keyword using a heuristic coarse classification mechanism, it can be subjectively categorized into a preset category by human judgment. This ensures that each word is assigned a corresponding category, avoiding the possibility of omitting any words from the list.

[0042] According to exemplary embodiments of this disclosure, multiple words can also be filtered using at least one filtering rule to obtain words retained after filtering. The at least one filtering rule may include at least one of the following: the frequency of the word is greater than or equal to a preset frequency threshold; the length of the word meets a preset length requirement; the word is not a preset stop word; the word is not a pure string of numbers; the word is not a pure string of symbols; and the number of words retained after filtering is less than or equal to a preset quantity threshold. For example, the preset quantity threshold may be, but is not limited to, 300.

[0043] Then, for each word retained after filtering, a heuristic coarse classification can be performed, that is, the word can be compared with multiple preset anchor keywords. In other words, in this disclosure, multiple words segmented from the target text can be filtered according to at least one filtering rule to filter out invalid words that do not meet the standards.

[0044] It should be noted that the aforementioned preset length requirement can be set according to actual needs. For example, the length of any word 't' can be set. The following conditions must be met: ; The aforementioned stop words may be words that have no practical significance in a specific field. For example, in the field of equipment quality, stop words may include but are not limited to: "he", "yu", "de", "le", "shi", etc.; the aforementioned pure numeric string may be composed of pure numbers, for example, it may be but not limited to: 1212345; the aforementioned pure symbol string may be composed of various symbols, wherein the various symbols may include but are not limited to punctuation marks and symbols such as @, #, ￥, %, &, ~. Illustratively, the pure symbol string may be but not limited to: , .! @#. In addition, in the present disclosure, words extracted from the target text also need to pass semantic consistency check to ensure that they have clear referentiality and domain adaptability in a specific context (e.g., equipment quality context).

[0045] In this way, by setting at least one filtering rule, invalid words that do not meet the standard can be filtered out, ensuring that only valid words with practical significance in a specific field are retained, thereby effectively avoiding that the constructed vocabulary contains too many invalid words.

[0046] According to an exemplary embodiment of the present disclosure, for each word among a plurality of words, when the word contains any one target preset anchor keyword under a target category, the target category corresponding to the target preset anchor keyword may be assigned to the word, wherein each preset category among a plurality of preset categories may correspond to at least one preset anchor keyword among the aforementioned plurality of preset anchor keywords, and the target category is a preset category among the plurality of preset categories.

[0047] Illustratively, the plurality of preset categories may be: equipment category, component category, phenomenon category, wherein the at least one preset anchor keyword corresponding to the equipment category may be: "motor", "pump", "fan"; the at least one preset anchor keyword corresponding to the component category may be: "bearing", "belt", "valve"; the at least one preset anchor keyword corresponding to the phenomenon category may be: "abnormal noise", "leakage", "heat generation".

[0048] It should be noted that when the same word hits multiple preset anchor keywords at the same time, multiple categories are allowed to be assigned to the word, instead of forcing uniqueness in the coarse classification stage. For example, assuming a certain word is "fan bearing", since the word contains both the preset anchor keyword "fan" under the equipment category and the preset anchor keyword "bearing" under the component category. Therefore, two categories, namely the equipment category and the component category, can be assigned to the word "fan bearing".

[0049] Additionally, if a word does not match any category, it can be manually categorized. For example, it can be assigned categories such as equipment, components, or phenomena based on user subjective judgment. Furthermore, if neither heuristic coarse classification nor manual categorization can assign a category to a word, it can be removed from the list.

[0050] In step 104, each word from the multiple words and the category assigned to each word can be input into the large language model.

[0051] It should be noted that the aforementioned words can be divided into several batches according to a preset batch size. Each batch can then be fed into the local large language model along with the words contained in each batch, their corresponding word frequencies, coarse classification results, and the JSON output specifications. For example, the preset batch size can be, but is not limited to, 40. The large language model can then perform noise removal, category correction, and alias normalization on each batch of input words. Furthermore, the prompts in the large language model can be set to explicitly require that the model can only perform category correction, alias normalization, and noise removal on words within the current batch, and cannot generate new words outside the current batch.

[0052] For example, assuming the number of words is N and the batch size is B, then the number of batches is the number of times the local large language model needs to be called. for: .

[0053] In step 105, the categories assigned to each word can be corrected using a large language model to obtain a vocabulary list, which can contain at least each word from multiple words and the corrected category corresponding to each word.

[0054] It should be noted that the JSON fields corresponding to words may have issues such as missing fields or incorrect value types. In such cases, the batch to which that word belongs will be considered an abnormal batch. In this situation, it is permissible to skip this abnormal batch and directly correct the category of words in the next batch. Furthermore, for problematic words in the abnormal batch, missing fields can be filled in manually, or incorrect field values ​​can be corrected. Then, the manually adjusted abnormal batch can be re-entered into the large language model for category correction. This process is iterated until no more abnormal batches appear.

[0055] According to exemplary embodiments of this disclosure, noise can also be removed from multiple words input to the large language model using a large language model to obtain words retained after noise removal. Noise removal may include at least one of the following: removing meaningless colloquialisms and removing generic words unrelated to the business theme expressed in the target text. Then, the categories assigned to the words retained after noise removal can be corrected using the large language model to obtain a vocabulary.

[0056] In this way, by using a large language model to remove noise from multiple words, we can eliminate meaningless colloquial words and general words that are irrelevant to the business theme expressed in the target text. This avoids the final vocabulary being mixed with too many meaningless words, and can improve the practicality and simplicity of the vocabulary.

[0057] According to exemplary embodiments of this disclosure, a mapping relationship between non-standard and standard expressions for the same word can also be obtained and output through a large language model. The non-standard expression may include at least one of the following: colloquial expression, abbreviation, slang, and incorrect expression. Furthermore, both the non-standard and standard expressions originate from multiple words.

[0058] For example, alias mapping The retention rules can be as follows:

[0059] That is, source word and target words Both must be valid strings, and the source word and target words At least one of them needs to be from the current batch.

[0060] In this way, by establishing a mapping relationship between non-standard and standard expressions of the same word through a large language model, the characteristics of the same word can be displayed more completely, making it easier for users to have a clearer and more comprehensive understanding of the word.

[0061] According to exemplary embodiments of this disclosure, for each word contained in the vocabulary, a large language model can be used to determine whether the category corresponding to the word is within a preset category whitelist, and / or whether the word originates from multiple words. It should be noted that determining whether a word originates from multiple words is to prevent hallucinations, which will be explained in detail later.

[0062] If the category corresponding to the word is within the preset category whitelist, and / or if the word originates from multiple words, the word and its corresponding category can remain in the word list; otherwise, the word and its corresponding category can be deleted from the word list.

[0063] It should be noted that the category whitelist can refer to a fixed set of allowed categories for output. Additionally, the vocabulary output by the large language model must satisfy the following joint constraints:

[0064] The retained word t must come from the current batch. .

[0065] It should be noted that while open-ended large language models possess strong semantic understanding capabilities, they are prone to generating illusory terms beyond the original text or exhibiting category drift in terminology extraction tasks. Therefore, this disclosure effectively suppresses illusory terms and category drift by setting source constraints and category whitelist constraints, thereby improving the reliability and verifiability of the results. Specifically, this disclosure introduces the local large language model into a constrained bootstrapping stage, explicitly requiring the model to complete category refinement, alias normalization, and noise removal only within the current batch, prohibiting the output of new terms outside the current batch. This approach utilizes the model's semantic understanding capabilities while avoiding uncontrolled expansion caused by open generation.

[0066] In addition, the large language model can record bootstrap metadata, such as, but not limited to, batch size, total number of batches, whether a local large language model is used, number of failed batches (abnormal batches), etc. Furthermore, the vocabulary and alias mapping table obtained above can be merged with existing base vocabulary or artificial seed vocabulary, and deduplication and sorting operations can be performed during the merging process. For example, words can be sorted according to their length; for instance, shorter words can be ranked higher in the list.

[0067] This disclosure enables cold-start discovery, category bootstrapping, alias normalization, and noise control of domain terms, even with only the original equipment quality issue text and in the absence of labeled corpora and mature domain dictionaries. This allows for the creation of a structured vocabulary resource that can be directly used for subsequent equipment quality text understanding, fact extraction, and knowledge graph construction. Furthermore, it reduces the risk of large language models introducing erroneous terms due to illusion. In addition, single-rule or single-step processing in related technologies often struggles to balance recall, accuracy, and interpretability, failing to directly output a sustainably expandable standard terminology and alias database. However, this disclosure, through a dual mechanism of "heuristic coarse classification + large language model category correction," ensures a vocabulary with higher accuracy and interpretability.

[0068] Figure 2 This is a flowchart illustrating a specific implementation of a vocabulary construction method according to exemplary embodiments of the present disclosure.

[0069] Reference Figure 2 In step 201, the original equipment quality problem text is obtained. This original equipment quality problem text usually comes from duty logs, inspection records, maintenance logs, and oral transcriptions, etc.

[0070] In step 202, the original equipment quality problem text is preprocessed to obtain preprocessed text. The preprocessing may include at least one of the following: removing punctuation marks from the original equipment quality problem text, and deleting spaces from the original equipment quality problem text.

[0071] In step 203, the preprocessed text is segmented using at least two word segmentation methods to obtain multiple words.

[0072] For example, words can be extracted from preprocessed text using the jieba word segmentation channel and the regular expression supplementary channel in related technologies. The jieba word segmentation channel is mainly used to extract regular words and phrases from the text, while the regular expression supplementary channel is mainly used to capture model numbers, abbreviations, equipment numbers, component abbreviations, and common phrase expressions in field records from the text.

[0073] In step 204, a category is assigned to each of the multiple words using a heuristic coarse classification mechanism.

[0074] For example, for each of the multiple words, the word can be compared with multiple preset anchor keywords, and if the word matches the target preset anchor keyword among the multiple preset anchor keywords, the word can be assigned a category corresponding to the target preset anchor keyword. The preset anchor keywords can refer to a set of prior keywords used for heuristically determining the category of a word.

[0075] In step 205, each word from the multiple words and the category assigned to each word are input into the large language model.

[0076] For example, the aforementioned words can be divided into several batches according to a preset batch size, and the words contained in each batch, the word frequency corresponding to the words, the coarse classification results of the words, and the JSON output specifications of the words can be sent to the local large language model one batch at a time.

[0077] In step 206, the categories assigned to each word are corrected using a large language model to obtain a vocabulary list, wherein the vocabulary list may contain at least each word from multiple words and the corrected category corresponding to each word.

[0078] Figure 3 This is a block diagram illustrating a vocabulary construction apparatus 300 according to an exemplary embodiment of the present disclosure.

[0079] Reference Figure 3 The vocabulary building device 300 may include a text acquisition module 301, a word segmentation module 302, a category assignment module 303, an input module 304, and a category correction module 305.

[0080] The text acquisition module 301 can acquire target text. For example, it can acquire original equipment quality problem text, which usually comes from duty logs, inspection records, maintenance logs, and oral transcriptions, etc.

[0081] The word segmentation module 302 can perform word segmentation on the target text to obtain multiple words. For example, it can automatically extract words or phrases that have practical significance in the field of equipment quality from the aforementioned original text on equipment quality issues.

[0082] According to exemplary embodiments of the present disclosure, the vocabulary construction apparatus 300 may further include a preprocessing module.

[0083] The preprocessing module can first preprocess the target text to obtain preprocessed text. This preprocessing can include at least one of the following: removing punctuation marks from the target text, and deleting spaces from the target text. Then, the word segmentation module 302 can perform word segmentation on the preprocessed text to obtain multiple words. That is, in this disclosure, before performing word segmentation on the target text, preprocessing such as symbol cleaning and whitespace normalization can be performed to make the text format more standardized and normalized, thus laying a good foundation for the subsequent word segmentation stage.

[0084] According to an exemplary embodiment of this disclosure, the word segmentation module 302 can segment the target text using at least two word segmentation methods to obtain multiple words. For example, words can be extracted from the target text using the jieba word segmentation channel and the regular expression supplementary channel from related technologies. The jieba word segmentation channel is mainly used to extract regular words and phrases from the text; the regular expression supplementary channel is mainly used to capture model numbers, abbreviations, equipment numbers, component abbreviations, and common phrase expressions found in field records from the text.

[0085] For each of the multiple words, the category assignment module 303 can compare the word with multiple preset anchor keywords. If the word matches a target preset anchor keyword among the multiple preset anchor keywords, it can assign the word a category corresponding to the target preset anchor keyword. Here, preset anchor keywords can refer to a set of prior keywords used for heuristically determining the category of a word. For example, in the field of device quality, the category of a word can include, but is not limited to: equipment, component, environment, phenomenon, action, result, cause, parameter, etc.

[0086] According to exemplary embodiments of the present disclosure, the vocabulary construction apparatus 300 may further include a filtering module.

[0087] The filtering module can also filter multiple words using at least one filtering rule to obtain the words retained after filtering. The at least one filtering rule may include at least one of the following: the frequency of the word is greater than or equal to a preset frequency threshold; the length of the word meets a preset length requirement; the word is not a preset stop word; the word is not a pure string of numbers; the word is not a pure string of symbols; and the number of words retained after filtering is less than or equal to a preset quantity threshold. For example, the preset quantity threshold may be, but is not limited to, 300.

[0088] Then, for each word retained after filtering, the category assignment module 303 can perform a heuristic coarse classification for that word, that is, compare the word with multiple preset anchor keywords.

[0089] According to an exemplary embodiment of the present disclosure, for each of a plurality of words, if the word contains any one of the target preset anchor keywords under the target category, the category allocation module 303 can assign the target category corresponding to the target preset anchor keyword to the word. Each of the plurality of preset categories can correspond to at least one of the aforementioned plurality of preset anchor keywords, and the target category can be a preset category among the plurality of preset categories.

[0090] Input module 304 can input each word from a plurality of words, along with the category assigned to each word, into the large language model.

[0091] The category correction module 305 can correct the category assigned to each word through a large language model to obtain a vocabulary, wherein the vocabulary can contain at least each word in multiple words and the corrected category corresponding to each word.

[0092] According to exemplary embodiments of the present disclosure, the vocabulary construction apparatus 300 may further include a noise removal module.

[0093] The noise removal module can remove noise from multiple words input to the large language model, obtaining the words retained after noise removal. Noise removal may include at least one of the following: removing meaningless colloquialisms, and removing generic words unrelated to the business theme expressed in the target text. Then, the category correction module 305 can correct the categories assigned to the words retained after noise removal using the large language model, obtaining a vocabulary.

[0094] According to an exemplary embodiment of this disclosure, the vocabulary construction apparatus 300 may further include a mapping relationship acquisition module.

[0095] The mapping relationship acquisition module can acquire and output the mapping relationship between non-standard and standard expressions for the same word through a large language model. The non-standard expression can include at least one of the following: colloquial expression, abbreviation, slang, and incorrect expression. Furthermore, both the non-standard and standard expressions are derived from multiple words.

[0096] According to exemplary embodiments of the present disclosure, the vocabulary construction apparatus 300 may further include a judgment module, a maintenance module, and a deletion module.

[0097] For each word in the vocabulary, the judgment module can use a large language model to determine whether the category corresponding to the word is within a preset category whitelist, and / or whether the word originates from multiple words.

[0098] If the category corresponding to the word is within the preset category whitelist, and / or the word originates from multiple words, the maintenance module can maintain the word and its corresponding category within the word list; otherwise, the deletion module can delete the word and its corresponding category from the word list.

[0099] Figure 4 This is a block diagram illustrating an electronic device 400 according to an exemplary embodiment of the present disclosure.

[0100] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402. The at least one memory 401 stores instructions that, when executed by the at least one processor 402, perform a vocabulary construction method according to an exemplary embodiment of the present disclosure.

[0101] As an example, electronic device 400 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 400 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 400 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0102] In electronic device 400, processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0103] The processor 402 can execute instructions or code stored in the memory 401, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0104] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 401 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 401 and the processor 402 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 402 to read files stored in the memory.

[0105] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 400 can be interconnected via a bus and / or network.

[0106] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the above-described vocabulary construction method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0107] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements the vocabulary construction method according to the present disclosure.

[0108] According to the lexicon construction method, apparatus, electronic device, storage medium, and computer program product disclosed herein, compared to the methods in related technologies that rely on manual construction of industry dictionaries, this disclosure provides an automated lexicon construction method, thereby effectively reducing the workload of manual lexicon maintenance and repeated word addition, and thus reducing maintenance costs. Furthermore, compared to the methods in related technologies that rely on supervised named entity recognition or relation extraction models for lexicon construction, this disclosure does not require a large-scale, high-quality labeled corpus; instead, it can initiate lexicon construction using only the original equipment quality problem text, reducing reliance on prior dictionaries and labeled data, thus enabling rapid deployment and making it more suitable for cold-start scenarios and small-sample scenarios involving equipment quality data.

[0109] According to an exemplary embodiment of this disclosure, before segmenting the target text, preprocessing such as symbol cleaning and whitespace normalization can be performed on the target text to make the text format more standardized and normalized, thereby laying a good foundation for the subsequent segmentation stage.

[0110] According to exemplary embodiments of this disclosure, by employing a word extraction mechanism that combines the jieba segmentation channel and the regular expression supplementation channel, a high-recall word set can be obtained from colloquial and abbreviated equipment quality problem texts without relying on mature professional dictionaries. Furthermore, the dual-channel candidate word discovery mechanism provided by this disclosure can effectively improve the coverage of colloquial expressions, abbreviations, slang, and erroneous descriptions. Compared to methods that rely entirely on manual vocabularies and rule templates, it can effectively reduce the workload of manual item-by-item maintenance and repeated word supplementation.

[0111] According to an exemplary embodiment of this disclosure, by constructing a heuristic coarse classification mechanism using preset anchor keywords, words can be mapped to the corresponding categories under unlabeled conditions, thereby providing prior constraints for subsequent refined bootstrapping.

[0112] According to exemplary embodiments of this disclosure, it can be ensured that each word is assigned a corresponding category, avoiding the phenomenon of omitting the assignment of categories to some words.

[0113] According to an exemplary embodiment of this disclosure, by setting at least one filtering rule, invalid words that do not meet the standards can be filtered out, so as to ensure that only valid words that have practical significance in a specific field are retained, thereby effectively avoiding the inclusion of too many invalid words in the constructed vocabulary.

[0114] According to an exemplary embodiment of this disclosure, by utilizing a large language model to remove noise from multiple words, it is possible to remove meaningless colloquial words and general words that are irrelevant to the business theme expressed by the target text, thereby avoiding the final vocabulary list from being mixed with too many meaningless words and improving the practicality and simplicity of the vocabulary list.

[0115] According to exemplary embodiments of this disclosure, establishing a mapping relationship between non-standard and standard expressions of the same word through a large language model can more completely display the characteristics of the same word, making it easier for users to have a clearer and more comprehensive understanding of the word.

[0116] According to exemplary embodiments of this disclosure, while open-ended large language models possess strong semantic understanding capabilities, they are prone to generating illusory terms beyond the original text or experiencing category drift in terminology extraction tasks. Therefore, this disclosure effectively suppresses the generation of illusory terms and category drift by setting source constraints and category whitelist constraints, thereby improving the reliability and verifiability of the results. Specifically, by introducing the local large language model into a constrained bootstrapping stage, explicitly requiring the model to complete category refinement, alias normalization, and noise removal only within the current batch, and prohibiting the output of new terms outside the current batch, the uncontrolled expansion caused by open generation can be avoided while utilizing the model's semantic understanding capabilities.

[0117] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0118] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A vocabulary construction method, characterized in that, include: Get the target text; The target text is segmented to obtain multiple words; For each of the multiple words, the word is compared with multiple preset anchor keywords, and if the word matches the target preset anchor keyword among the multiple preset anchor keywords, the word is assigned the category corresponding to the target preset anchor keyword; Input each of the multiple words and the category assigned to each word into the large language model; The large language model is used to correct the categories assigned to each word to obtain a vocabulary list, wherein the vocabulary list contains at least each of the plurality of words and the corrected category corresponding to each word.

2. The vocabulary construction method as described in claim 1, characterized in that, Before performing word segmentation on the target text to obtain multiple words, the process also includes: The target text is preprocessed to obtain preprocessed text, wherein the preprocessing includes at least one of the following: removing punctuation marks contained in the target text, and deleting spaces contained in the target text; The target text is segmented to obtain multiple words, including: The preprocessed text is segmented to obtain the multiple words.

3. The vocabulary construction method as described in claim 1, characterized in that, The target text is segmented to obtain multiple words, including: The target text is segmented using at least two different word segmentation methods to obtain the multiple words.

4. The vocabulary construction method as described in claim 1, characterized in that, Before comparing each of the multiple words with multiple preset anchor keywords, the process also includes: The plurality of words are filtered using at least one filtering rule to obtain words that are retained after filtering. The at least one filtering rule includes at least one of the following: the frequency of the word is greater than or equal to a preset frequency threshold, the length of the word meets a preset length requirement, the word is not a preset stop word, the word is not a pure string of numbers, the word is not a pure string of symbols, and the number of words retained after filtering is less than or equal to a preset number threshold. The step of comparing each of the plurality of words with a plurality of preset anchor keywords includes: For each word retained after filtering, compare it with the plurality of preset anchor keywords.

5. The vocabulary construction method as described in claim 1, characterized in that, For each of the plurality of words, the word is compared with a plurality of preset anchor keywords, and if the word matches a target preset anchor keyword among the plurality of preset anchor keywords, the word is assigned a category corresponding to the target preset anchor keyword, including: For each of the plurality of words, if the word contains any one of the target preset anchor keywords under the target category, the target category corresponding to the target preset anchor keyword is assigned to the word. Each of the plurality of preset categories corresponds to at least one preset anchor keyword among the plurality of preset anchor keywords, and the target category is a preset category among the plurality of preset categories.

6. The vocabulary construction method as described in claim 1, characterized in that, Also includes: The large language model is used to obtain and output the mapping relationship between non-standard and standard expressions for the same word, wherein the non-standard expression includes at least one of the following: colloquial expression, abbreviation, slang, and incorrect expression, and both the non-standard expression and the standard expression are derived from the multiple words.

7. The vocabulary construction method as described in claim 1, characterized in that, Before obtaining the vocabulary by correcting the categories assigned to each word using the large language model, the following steps are also included: The large language model is used to remove noise from multiple words input to the large language model to obtain the words retained after noise removal. The noise removal includes at least one of the following: removing meaningless colloquial words and removing general words that are irrelevant to the business theme expressed by the target text. The step of correcting the category assigned to each word using the large language model to obtain a vocabulary includes: The vocabulary is obtained by correcting the categories assigned to the words retained after noise removal using the large language model.

8. The vocabulary construction method as described in claim 1, characterized in that, After correcting the categories assigned to each word using the large language model to obtain the vocabulary, the process also includes: For each word contained in the vocabulary, the large language model determines whether the category corresponding to the word is within a preset category whitelist, and / or whether the word originates from the multiple words; If the category corresponding to the word is within the preset category whitelist, and / or if the word originates from multiple words, the word and its corresponding category shall remain within the word list; Otherwise, the word and its corresponding category will be deleted from the word list.

9. A vocabulary construction apparatus, characterized in that, include: The text acquisition module is configured to acquire the target text. The word segmentation module is configured to perform word segmentation on the target text to obtain multiple words; The category assignment module is configured to compare each of the plurality of words with a plurality of preset anchor keywords, and assign the word to the category corresponding to the target preset anchor keyword if the word matches the target preset anchor keyword among the plurality of preset anchor keywords; The input module is configured to input each of the plurality of words and the category assigned to each word into the large language model; The category correction module is configured to correct the category assigned to each word using the large language model to obtain a vocabulary, wherein the vocabulary contains at least each of the plurality of words and the corrected category corresponding to each word.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the vocabulary construction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the vocabulary construction method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the vocabulary construction method as described in any one of claims 1 to 8.

Images