Confusion word mining method and device of large language model LLM and storage medium

Abstract

The invention discloses a confusion word mining method and device of a large language model LLM and a storage medium, and relates to the technical field of smart home / smart home, the confusion word mining method of the large language model LLM comprises the following steps: obtaining first text information; under the condition that at least one candidate confusion word of the first text word is determined, replacing the first text word in the first text information with the candidate confusion word to obtain second text information; inputting the first text information and the second text information into LLM, and indicating the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information; and determining that the candidate confusion word and the first text word belong to the same confusion word pair and determining that the candidate confusion word and the first text word belong to the same confusion word pair under the condition that the first recognition result and the second recognition result are different.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a method, device and storage medium for mining confusing words of a large language model (LLM). Background Art

[0002] The related art usually uses a traditional confusion dictionary or a separately trained confusion word recognition model to mine confusion words in a large language model (LLM). However, if the confusion dictionary is not updated in time, the accuracy of the confusion words will be affected, and the separately trained confusion word recognition model will increase the cost. In addition, the application of the confusion word recognition model is relatively single. After high-cost training, it can usually only be used to recognize confusion words, which is low in cost performance. Therefore, the use of a separately trained confusion word recognition model cannot be used as an effective way to improve the accuracy of obtaining confusion words. That is, the related art has a technical problem of low accuracy in the mining of confusion words in a large language model (LLM). Summary of the invention

[0003] The embodiments of the present application provide a method, device, storage medium and electronic device for mining confusion words of a large language model (LLM), so as to at least solve the technical problem of low accuracy in mining confusion words of a large language model (LLM).

[0004] According to one aspect of an embodiment of the present application, a method for mining confusion words of a large language model (LLM) is provided, comprising: obtaining first text information, wherein the first text information includes a first text word; when at least one candidate confusion word of the first text word is determined, replacing the first text word in the first text information with the candidate confusion word to obtain second text information; inputting the first text information and the second text information into the LLM, and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used to recognize text information; when there is a difference between the first recognition result and the second recognition result, determining that the candidate confusion word and the first text word belong to the same confusion word pair.

[0005] According to another aspect of an embodiment of the present application, a confusion word mining device for a large language model (LLM) is provided, comprising: a first acquisition unit, configured to acquire first text information, wherein the first text information includes a first text word; a first replacement unit, configured to, when at least one candidate confusion word of the first text word is determined, replace the first text word in the first text information with the candidate confusion word to obtain second text information; a first recognition unit, configured to input the first text information and the second text information into the LLM, and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used to recognize text information; and a first determination unit, configured to, when there is a difference between the first recognition result and the second recognition result, determine that the candidate confusion word and the first text word belong to the same confusion word pair.

[0006] As an optional solution, the apparatus further includes: a first input unit, configured to input the first text word into the LLM after acquiring the first text information, and instruct the LLM to output a confusing word to obtain the candidate confusing word.

[0007] As an optional solution, the device further includes: a second input unit for inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information; if the first recognition result indicates that the first text information is incorrect in meaning and the second recognition result also indicates that the second text information is incorrect in meaning, inputting the first text word into the LLM and instructing the LLM to output a confusing word other than the candidate confusing word to obtain a new confusing word; and a second replacement unit for inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, using the new confusing word to replace the first text word. The method comprises the steps of: replacing the first text word in the first text information with the first text word to obtain new text information; a second recognition unit, which is used to input the first text information and the second text information into the LLM and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, and then input the new text information into the LLM and instruct the LLM to perform semantic recognition to obtain a third recognition result corresponding to the new text information; and a second determination unit, which is used to input the first text information and the second text information into the LLM and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, and when the third recognition result indicates that the new text information is correct in meaning, determine that the new confusing word and the first text word belong to the same confusing word pair.

[0008] As an optional solution, the above-mentioned device also includes: an adding unit, which is used to add the above-mentioned candidate confused words and the above-mentioned first text word to a confused word library after it is determined that the above-mentioned candidate confused words and the above-mentioned first text word belong to the same confused word pair, wherein the above-mentioned confused word library is used to store multiple text words with a confusing relationship, and the above-mentioned candidate confused words and the above-mentioned first text word determined to be the same confused word pair are regarded as having the above-mentioned confusing relationship.

[0009] As an optional solution, the device further includes: a second acquisition unit, which is used to acquire third text information after adding the candidate confused words and the first text word to the confused word library, wherein the third text information includes the second text word; a third acquisition unit, which is used to determine the third text word having the confusion relationship with the second text word from the confused word library after adding the candidate confused words and the first text word to the confused word library, if it is recognized that the meaning of the third text information is incorrect; and a fourth acquisition unit, which is used to replace the second text word in the third text information with the third text word to obtain fourth text information after adding the candidate confused words and the first text word to the confused word library.

[0010] As an optional solution, the above-mentioned device also includes: a third determination unit, which is used to, after the above-mentioned first text information and the above-mentioned second text information are input into the LLM and the above-mentioned LLM is instructing to perform semantic recognition to obtain a first recognition result corresponding to the above-mentioned first text information and a second recognition result corresponding to the above-mentioned second text information, determine that there is a difference between the first recognition result and the second recognition result when the above-mentioned first recognition result indicates that the meaning of the above-mentioned first text information is correct and the above-mentioned second recognition result indicates that the meaning of the above-mentioned second text information is incorrect, or when the above-mentioned first recognition result indicates that the meaning of the above-mentioned first text information is incorrect and the above-mentioned second recognition result indicates that the meaning of the above-mentioned second text information is correct; or, when the above-mentioned first recognition result indicates that the meaning of the above-mentioned first text information is correct and the above-mentioned second recognition result also indicates that the meaning of the above-mentioned second text information is correct, determine that the above-mentioned first text information and the above-mentioned second text information belong to a confusing sentence pair.

[0011] As an optional solution, the above-mentioned device also includes: a fifth acquisition unit, which is used to acquire at least one of the following auxiliary identification information before inputting the above-mentioned first text information and the above-mentioned second text information into the LLM and instructing the above-mentioned LLM to perform semantic recognition to obtain the first recognition result corresponding to the above-mentioned first text information and the second recognition result corresponding to the above-mentioned second text information: domain information of the field to which the above-mentioned first text information belongs, context information of the information content of the above-mentioned first text information; the above-mentioned first recognition unit includes: a recognition module, which is used to input the above-mentioned auxiliary recognition information, the above-mentioned first text information and the above-mentioned second text information into the above-mentioned LLM, and instruct the above-mentioned LLM to perform semantic recognition to obtain the above-mentioned first recognition result and the above-mentioned second recognition result.

[0012] According to another aspect of the embodiment of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above large language model LLM confusion word mining method.

[0013] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-mentioned method for mining confused words of the large language model LLM through the computer program.

[0014] In an embodiment of the present application, first text information is obtained, wherein the first text information includes a first text word; when at least one candidate confusion word of the first text word is determined, the candidate confusion word is used to replace the first text word in the first text information to obtain second text information; the first text information and the second text information are input into an LLM, and the LLM is instructed to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used to recognize text information; when there is a difference between the first recognition result and the second recognition result, it is determined that the candidate confusion word and the first text word belong to the same confusion word pair. When at least one candidate confused word of the first text word is determined, the accuracy of the candidate confused word cannot be guaranteed, and the use of a separately trained confused word recognition model will increase the confused word mining cost of the large language model LLM. The more general large language model is further used to perform semantic recognition on the original text information and the text information with the candidate confused words, and then the accuracy of the candidate confused words is indirectly judged by the semantic recognition result output by the large language model, thereby achieving the purpose of obtaining a confused word with high accuracy without separately training a confused word recognition model, thereby achieving the technical effect of improving the accuracy of the confused word mining of the large language model LLM, and thus solving the technical problem of low accuracy of the confused word mining of the large language model LLM. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Figure 1 is a schematic diagram of a hardware environment of an interaction method of a smart device according to an embodiment of the present application;

[0018] Figure 2 It is a schematic diagram of the process of an optional large language model LLM confusion word mining method according to an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of an optional method for mining confusing words of a large language model LLM according to an embodiment of the present application;

[0020] Figure 4 It is a schematic diagram of another optional method for mining confusing words of a large language model LLM according to an embodiment of the present application;

[0021] Figure 5 It is a schematic diagram of an optional confusion word mining device of a large language model LLM according to an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of another optional device for mining confused words of a large language model LLM according to an embodiment of the present application;

[0023] Figure 7 is a schematic diagram of another optional device for mining confused words of a large language model LLM according to an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of another optional device for mining confused words of a large language model LLM according to an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of understanding, the following terms are explained:

[0029] LLM: full name (Large Language Model) Large Language Model. The recently appeared ChatGPT, GPT4, Wenxin Yiyan, Tongyi Qianwen are all implementations of large languages. The calculation speed and resource consumption of LLM increase with the increase of input length. The calculation speed decreases and the resource consumption increases.

[0030] Asr: speech to text.

[0031] ASR error correction: Correct the errors in the text obtained by voice-to-text conversion.

[0032] Confused words: words that are easily misidentified during ASR recognition, for example, "shoes" is easily recognized as "writing". The confused word dictionary is an indispensable part of the ASR error correction module. High-quality confused words are the guarantee of the ASR error correction module.

[0033] Prompt: An instruction template that needs to be constructed when using LLM so that LLM can better answer questions.

[0034] According to one aspect of an embodiment of the present application, a method for interacting with a smart home device is provided. The method for interacting with a smart home device is widely used in smart home (Smart Home), smart home, smart home device ecosystem, smart home (IntelligenceHouse) ecosystem and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the method for interacting with the smart home device can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0035] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0036] Optionally, as an optional implementation, as Figure 2 As shown in the figure, the confusion word mining method of the large language model LLM includes:

[0037] S202, obtaining first text information, wherein the first text information includes a first text word;

[0038] S204, when at least one candidate confusing word of the first text word is determined, the first text word in the first text information is replaced with the candidate confusing word to obtain second text information;

[0039] S206, inputting the first text information and the second text information into the LLM, and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used for recognizing text information;

[0040] S208: When there is a difference between the first recognition result and the second recognition result, determine that the candidate confusing word and the first text word belong to the same confusing word pair.

[0041] Optionally, in this embodiment, the above-mentioned large language model LLM confusion word mining method can be applied to, but not limited to, the confusion word mining scenario. First, the initial confusion word is obtained by the traditional confusion word mining method, and then the more general LLM is used for semantic recognition, and then based on the semantic recognition result, the initial confusion word is indirectly judged for accuracy to ensure the accuracy of the mined confusion word. Among them, since a more general large language model is used instead of a separately trained confusion word recognition model, even if a large language model is trained using certain sample resources, the large language model can be cited in many fields, and is not limited to the field of confusion word recognition, and the cost performance is high. Therefore, this embodiment can be used as an effective way to improve the efficiency of the large language model LLM confusion word mining.

[0042] Optionally, in the present embodiment, the confusion word mining can be understood as, but not limited to, extracting similar words or expressions in the text database to replace the first text word, so as to achieve the purpose of confusing or concealing the true intention of the first text word. The method of determining the candidate confusion words of the first text word can also be understood as, but not limited to, a method of mining confusion words, such as synonym mining, finding synonyms or near synonyms with similar meanings to the keyword, and finding words that may be used for confusion by comparing the meaning, semantics, part of speech and other features. Synonym mining can be performed through tools such as professional dictionaries, synonym dictionaries, and corpora. Another example is antonym mining, finding words with opposite meanings to the keyword, which are used to replace the opposite meanings of the keyword. Antonym mining can be performed through tools such as antonym dictionaries and corpora. Another example is fuzzy word mining, finding words with similar meanings to the keyword but with vague meanings, so that readers are easily confused or misunderstood when understanding. Fuzzy word mining can be performed through corpora, relevant documents and other channels. Among them, the number of candidate confusion words can be one or more, which is only illustrated here, and the number is not limited.

[0043] Optionally, in this embodiment, the first text word in the first text information is replaced by a candidate confusing word to obtain the second text information, such as Figure 3 As shown, the first text information 302 includes the text information "this shoe is dirty", wherein the text information "shoe" is the first text word 304, and then the candidate confusion word 306 "writing" replaces the first text word 304, thereby obtaining the second text information 308 "this writing is dirty".

[0044] Optionally, in this embodiment, a large language model (LLM) is a neural network model obtained by training with multiple samples and used to recognize text information, such as chatgpt, gpt4, etc., wherein the calculation speed and resource consumption of LLM increase with the increase of input length, the calculation speed decreases and the resource consumption increases. LLM can realize a wide range of functions, involving semantic recognition, confusion word recognition, error correction recognition, etc.

[0045] Optionally, in this embodiment, the difference between the first recognition result and the second recognition result can be regarded as, but not limited to, that one text information is ambiguous, while the other text information is not ambiguous, which is consistent with the ambiguous distribution of confusing words in a sentence. Therefore, in this embodiment, when the first recognition result and the second recognition result are different, it is determined that the candidate confusing word and the first text word belong to the same confusing word pair. If either one of the first text information and the second text information is correct and the other text information is not correct, it can be regarded as that the first recognition result and the second recognition result are different.

[0046] To further illustrate, the recognition result can be optionally used, but not limited to, to represent the semantic content expressed by the text information. For example, the semantic content expressed by the text word "shoes" is a wearable daily item, while the semantic content expressed by the text word "writing" is the act of writing on paper or other media with a pen or other tool, and there is a clear difference between the two. Therefore, the first recognition result corresponding to the first text information "the shoes are dirty" indicates that the first text information is correct, while the second recognition result corresponding to the second text information "the writing is dirty" indicates that the second text information is incorrect, which indicates that there is a difference between the first recognition result and the second recognition result.

[0047] It should be noted that, when at least one candidate confused word for the first text word is determined, the accuracy of the candidate confused words cannot be guaranteed, and the use of a separately trained confused word identification model will increase the confused word mining cost of the large language model LLM. Further using a relatively general large language model, semantic recognition is performed on the original text information and the text information with the candidate confused words, and then the accuracy of the candidate confused words is indirectly judged by the semantic recognition result output by the large language model, thereby achieving the purpose of obtaining confused words with higher accuracy without the need to separately train a confused word identification model, thereby achieving the technical effect of improving the accuracy of confused word mining of the large language model LLM.

[0048] To further illustrate, the optional Figure 3 The scenario shown, continuing with e.g. Figure 4As shown, first text information 302 is obtained, wherein the first text information 302 includes a first text word 304; when a candidate confusing word 306 of the first text word 304 is determined, the first text word 304 in the first text information 302 is replaced with the candidate confusing word 306 to obtain second text information 308; the first text information 302 and the second text information 308 are input into a large language model 402, and the large language model 402 is instructed to perform semantic recognition, so as to obtain a first recognition result 404 corresponding to the first text information 302 and a first recognition result 406 corresponding to the second text information 308. The large language model 402 is a neural network model obtained by training with multiple samples and used to recognize text information; when the first recognition result 404 indicates that the meaning of the first text information 302 is correct and the second recognition result 406 indicates that the meaning of the second text information 308 is incorrect, or when the first recognition result 404 indicates that the meaning of the first text information 302 is incorrect and the second recognition result 406 indicates that the meaning of the second text information 308 is correct, it is determined that the candidate confused word 306 and the first text word 304 belong to the same confused word pair.

[0049] Through the embodiments provided by the present application, first text information is obtained, wherein the first text information includes a first text word; when at least one candidate confusing word of the first text word is determined, the first text word in the first text information is replaced by the candidate confusing word to obtain second text information; the first text information and the second text information are input into an LLM, and the LLM is instructed to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used to recognize text information; when there is a difference between the first recognition result and the second recognition result, it is determined that the candidate confusing word and the first text word belong to the same confusing word pair. When at least one candidate confused word of the first text word is determined, the accuracy of the candidate confused words cannot be guaranteed, and the use of a separately trained confused word recognition model will increase the confused word mining cost of the large language model LLM. The more general large language model is further used to perform semantic recognition on the original text information and the text information with the candidate confused words, and then the accuracy of the candidate confused words is indirectly judged by the semantic recognition result output by the large language model, thereby achieving the purpose of obtaining confused words with higher accuracy without separately training a confused word recognition model, thereby achieving the technical effect of improving the accuracy of confused word mining of the large language model LLM.

[0050] As an optional solution, after obtaining the first text information, the method further includes:

[0051] The first text word is input into the LLM, and the large language model is instructed to output the confusion word to obtain the candidate confusion word.

[0052] It should be noted that the large language model has high versatility and can also have the function of identifying confusing words. However, considering the cost factor, it is difficult to guarantee the accuracy of the large language model in identifying confusing words. Therefore, it is necessary to perform secondary recognition on the recognition result of the large language model for confusing words, that is, input the first text word into the LLM, and instruct the large language model to output the confusing word, and after obtaining the candidate confusing word, the candidate confusing word is recognized by the large language model. That is, the first text information and the second text information are input into the LLM, and the LLM is instructed to perform semantic recognition, and the first recognition result corresponding to the first text information and the second recognition result corresponding to the second text information are obtained.

[0053] Through the embodiment provided by the present application, the first text word is input into the LLM, and the large language model is instructed to output the confused word to obtain the candidate confused word, thereby achieving the purpose of improving the versatility of the large language model, thereby achieving the technical effect of improving the utilization rate of model resources.

[0054] As an optional solution, after inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, the method further includes:

[0055] S1-1, when the first recognition result indicates that the first text information has an incorrect meaning, and the second recognition result also indicates that the second text information has an incorrect meaning, input the first text word into the LLM, and instruct the large language model to output the confused words except the candidate confused words, to obtain a new confused word;

[0056] S1-2, using a new confusing word to replace the first text word in the first text information to obtain new text information;

[0057] S1-3, inputting new text information into the LLM, and instructing the LLM to perform semantic recognition to obtain a third recognition result corresponding to the new text information;

[0058] S1-4, when the third recognition result indicates that the new text information is correctly expressed, determining that the new confusing word and the first text word belong to the same confusing word pair.

[0059] It should be noted that, if the recognition result output by the large language model indicates that both the first text information and the second text information are incorrect, it will be regarded as a problem with the original candidate confused words, and the original candidate confused words will be discarded, and the large language model will be re-instructed to output confused words other than the candidate confused words to obtain new confused words, so as to improve the accuracy of the confused word mining of the large language model LLM.

[0060] Through the embodiments provided by the present application, when the first recognition result indicates that the meaning of the first text information is incorrect and the second recognition result also indicates that the meaning of the second text information is incorrect, the first text word is input into the LLM, and the large language model is instructed to output confused words other than candidate confused words to obtain new confused words; the new confused words are used to replace the first text word in the first text information to obtain new text information; the new text information is input into the LLM, and the LLM is instructed to perform semantic recognition to obtain a third recognition result corresponding to the new text information; when the third recognition result indicates that the meaning of the new text information is correct, it is determined that the new confused word and the first text word belong to the same confused word pair, thereby achieving the technical effect of improving the accuracy of confused word mining of the large language model LLM.

[0061] As an optional solution, after determining that the candidate confusing word and the first text word belong to the same confusing word pair, the method further includes:

[0062] The candidate confusing words and the first text word are added to a confusing word library, wherein the confusing word library is used to store multiple text words with a confusing relationship, and the candidate confusing words and the first text word determined to be the same confusing word pair are considered to have a confusing relationship.

[0063] Optionally, in this embodiment, the confusion word library can be understood as, but not limited to, a database for dynamically updating confused words. When performing confused word mining, corresponding confused words are preferentially extracted from the confusion word library. When the corresponding confused words are missing from the confusion word library, or the confused words stored in the confusion word library do not meet the timeliness condition, the confused word mining method of the large language model LLM is executed to dynamically update the confused words.

[0064] It should be noted that in order to facilitate the use of confusion words, a confusion word library will be set up, and the text words added to the confusion word library will be limited to text words recognized by speech of a large language model, such as adding candidate confusion words and the first text word belonging to the same confusion word pair to the confusion word library.

[0065] Through the embodiments provided by the present application, the candidate confusing words and the first text word are added to the confusing word library, wherein the confusing word library is used to store multiple text words having a confusing relationship, and the candidate confusing words and the first text word determined to be the same confusing word pair are considered to have a confusing relationship, thereby achieving the purpose of providing convenience in the use of confusing words, thereby achieving the technical effect of improving the efficiency of using confusing words.

[0066] As an optional solution, after adding the candidate confusing words and the first text word to the confusing word library, the method further includes:

[0067] S2-1, obtaining third text information, wherein the third text information includes the second text word;

[0068] S2-2, when it is identified that the third text information has an incorrect meaning, determining a third text word that has a confusing relationship with the second text word from a confusing word library;

[0069] S2-3, using the third text word to replace the second text word in the third text information to obtain fourth text information.

[0070] It should be noted that, in the actual use of confusing words, if it is identified that the third text information is incorrect in meaning, the corresponding confusing words are determined from the confusing word library to assist semantic recognition and improve the success rate of semantic recognition.

[0071] Through the embodiments provided by the present application, third text information is obtained, wherein the third text information includes a second text word; when it is identified that the third text information is incorrectly expressed, a third text word having a confusing relationship with the second text word is determined from a confusion word library; the third text word is used to replace the second text word in the third text information to obtain fourth text information, thereby achieving the purpose of assisting semantic recognition through the confusion word library, thereby realizing the technical effect of improving the success rate of semantic recognition.

[0072] As an optional solution, after inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, the method further includes:

[0073] When the first recognition result indicates that the meaning of the first text information is correct and the second recognition result indicates that the meaning of the second text information is incorrect, or when the first recognition result indicates that the meaning of the first text information is incorrect and the second recognition result indicates that the meaning of the second text information is correct, determining that there is a difference between the first recognition result and the second recognition result;

[0074] When the first recognition result indicates that the meaning of the first text information is correct and the second recognition result also indicates that the meaning of the second text information is correct, it is determined that the first text information and the second text information belong to a confusing sentence pair.

[0075] It should be noted that, if the first recognition result indicates that the meaning of the first text information is correct, and the second recognition result also indicates that the meaning of the second text information is correct, it will be regarded as that the first text information and the second text information themselves are confused, and then the confused recognition of text words needs to be placed after the confused recognition of text sentences. Therefore, it is further determined that the first text information and the second text information belong to a confusing sentence pair, thereby improving the processing efficiency of text information.

[0076] Through the embodiments provided by the present application, when the first recognition result indicates that the meaning of the first text information is correct and the second recognition result also indicates that the meaning of the second text information is correct, it is determined that the first text information and the second text information belong to a confusing sentence pair, thereby achieving the technical effect of improving the processing efficiency of text information.

[0077] As an optional solution, before inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, the method further includes: obtaining at least one of the following auxiliary recognition information: domain information of the domain to which the first text information belongs, and context information of the information content in which the first text information is located;

[0078] As an optional solution, the first text information and the second text information are input into LLM, and LLM is instructed to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, including: inputting auxiliary recognition information, the first text information and the second text information into LLM, and instructing LLM to perform semantic recognition to obtain a first recognition result and a second recognition result.

[0079] It should be noted that, in order to improve the accuracy of semantic recognition, the auxiliary recognition information, the first text information and the second text information are input into the LLM, and the semantic recognition result with higher accuracy will provide an accurate basis for the judgment of confusing word pairs.

[0080] Through the embodiments provided in the present application, at least one of the following auxiliary identification information is obtained: domain information of the field to which the first text information belongs, context information of the information content in which the first text information is located; the auxiliary identification information, the first text information, and the second text information are input into the LLM, and the LLM is instructed to perform semantic recognition to obtain a first recognition result and a second recognition result, thereby achieving the technical effect of improving the accuracy of semantic recognition.

[0081] As an optional solution, for ease of understanding, the above-mentioned large language model LLM's confusion word mining method is applied to the confusion word mining scenario in the vertical field. Specifically, firstly, the existing field data is retrieved and databased, and then the existing confusion word mining and LLM are used directly to obtain candidate confusion word pairs, such as a1 and a2, where a1 corresponds to sentence q1 and a2 corresponds to sentence q2; and sentence q1 and sentence q2 obtained after replacing the confusion word are used to retrieve the greeting field data; further, the field data, the previous sentence, q1, and q2 are combined to construct a prompt input into the LLM, and the LLM is asked to output the correctness of q1 and q2. One of the four combinations may be obtained. If q1 and q2 are both correct, it means that the sentence itself is ambiguous and no confusion word is generated. If both are wrong, a new a3 and a new q3 are generated, and the above steps are repeated. In other cases, the accuracy check of the confusion word is considered to have passed.

[0082] Through the embodiments provided in this application, a vertical domain confusion word mining framework based on LLM is implemented, and this framework does not need to fine-tune the LLM model in the vertical domain. Through high-quality confusion words, the quality of ASR error correction can be quickly improved without changing the existing ASR error correction module. At the same time, context information is utilized and domain knowledge is introduced. The entire process does not require human intervention and is fully automated to eliminate ambiguity.

[0083] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0084] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0085] According to another aspect of the embodiment of the present application, a device for mining confused words of a large language model LLM for implementing the above-mentioned method for mining confused words of a large language model LLM is also provided. Figure 5 As shown, the device comprises:

[0086] A first acquisition unit 502 is used to acquire first text information, wherein the first text information includes a first text word;

[0087] A first replacement unit 504 is configured to, when at least one candidate confusing word of the first text word is determined, replace the first text word in the first text information with the candidate confusing word to obtain second text information;

[0088] A first recognition unit 506 is used to input the first text information and the second text information into the LLM, and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used for recognizing text information;

[0089] The first determining unit 508 is configured to determine that the candidate confusing word and the first text word belong to the same confusing word pair when there is a difference between the first recognition result and the second recognition result.

[0090] For a specific embodiment, reference may be made to the example shown in the confusion word mining device of the large language model LLM, which will not be described in detail in this example.

[0091] As an alternative, Figure 6 As shown, the device also includes:

[0092] The first input unit 602 is used to input the first text word into the LLM after acquiring the first text information, and instruct the large language model to output the confusion word to obtain the candidate confusion word.

[0093] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for mining confusing words of the large language model LLM, which will not be described in detail in this example.

[0094] As an optional solution, the device also includes:

[0095] A second input unit is used for inputting the first text information and the second text information into the LLM, and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, and when the first recognition result indicates that the first text information has an incorrect meaning and the second recognition result also indicates that the second text information has an incorrect meaning, inputting the first text word into the LLM, and instructing the large language model to output a confused word other than a candidate confused word to obtain a new confused word;

[0096] A second replacement unit is used to replace the first text word in the first text information with a new confusing word to obtain new text information after inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information;

[0097] The second recognition unit is used to input the first text information and the second text information into the LLM, and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, and then input new text information into the LLM and instruct the LLM to perform semantic recognition to obtain a third recognition result corresponding to the new text information;

[0098] The second determination unit is used to input the first text information and the second text information into the LLM and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, and when a third recognition result indicates that the new text information is correct in terms of meaning, determine that the new confusing word and the first text word belong to the same confusing word pair.

[0099] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for mining confusing words of the large language model LLM, which will not be described in detail in this example.

[0100] As an alternative, Figure 7 As shown, the device also includes:

[0101] The adding unit 702 is used to add the candidate confusing word and the first text word to a confusing word library after determining that the candidate confusing word and the first text word belong to the same confusing word pair, wherein the confusing word library is used to store multiple text words with a confusing relationship, and the candidate confusing word and the first text word determined to be the same confusing word pair are considered to have a confusing relationship.

[0102] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for mining confusing words of the large language model LLM, which will not be described in detail in this example.

[0103] As an optional solution, the device also includes:

[0104] A second acquisition unit is used to acquire third text information after adding the candidate confusing word and the first text word to the confusing word library, wherein the third text information includes the second text word;

[0105] A third acquisition unit is used to determine a third text word having a confusing relationship with the second text word from the confusion word library after adding the candidate confusing word and the first text word to the confusion word library and in the case that the third text information is identified to have an incorrect meaning;

[0106] The fourth acquisition unit is used to replace the second text word in the third text information with the third text word after adding the candidate confusing word and the first text word to the confusing word library to obtain the fourth text information.

[0107] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for mining confusing words of the large language model LLM, which will not be described in detail in this example.

[0108] As an alternative, Figure 8 As shown, the device also includes:

[0109] The third determination unit 802 is used to determine that there is a difference between the first recognition result and the second recognition result when the first recognition result indicates that the meaning of the first text information is correct and the second recognition result indicates that the meaning of the second text information is incorrect, or when the first recognition result indicates that the meaning of the first text information is incorrect and the second recognition result indicates that the meaning of the second text information is correct; or, after the first text information and the second text information are input into the LLM and the LLM is instructed to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, when the first recognition result indicates that the meaning of the first text information is correct and the second recognition result also indicates that the meaning of the second text information is correct, determine that the first text information and the second text information belong to a confusing sentence pair.

[0110] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for mining confusing words of the large language model LLM, which will not be described in detail in this example.

[0111] As an optional solution, the device further includes: a fifth acquisition unit, which is used to acquire at least one of the following auxiliary recognition information before inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information: domain information of the field to which the first text information belongs, and context information of the information content in which the first text information is located;

[0112] The first recognition unit 506 includes: a recognition module, which is used to input the auxiliary recognition information, the first text information and the second text information into the LLM, and instruct the LLM to perform semantic recognition to obtain a first recognition result and a second recognition result.

[0113] For a specific embodiment, reference may be made to the example shown in the above-mentioned method for mining confusing words of the large language model LLM, which will not be described in detail in this example.

[0114] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned large language model LLM confusion word mining method is also provided. The electronic device can be but is not limited to the following: Fig. 9 As shown, it includes a memory 902 and a processor 904, the memory 902 stores a computer program, and the processor 904 is configured to execute the steps in any of the above method embodiments through the computer program.

[0115] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0116] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0117] S1, obtaining first text information, wherein the first text information includes a first text word;

[0118] S2, when at least one candidate confusing word of the first text word is determined, using the candidate confusing word to replace the first text word in the first text information to obtain second text information;

[0119] S3, inputting the first text information and the second text information into the LLM, and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used for recognizing text information;

[0120] S4: When there is a difference between the first recognition result and the second recognition result, determine that the candidate confusing word and the first text word belong to the same confusing word pair.

[0121] Alternatively, a person skilled in the art may understand that: Fig. 9 The structure shown is for illustration only. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig. 9 Different configurations are shown.

[0122] Among them, the memory 902 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for mining confusion words of the large language model LLM in the embodiment of the present application. The processor 904 executes various functional applications and data processing by running the software programs and modules stored in the memory 902, that is, to implement the above-mentioned method for mining confusion words of the large language model LLM. The memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 902 may further include a memory remotely arranged relative to the processor 904, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 902 can be specifically used, but not limited to, to store information such as the first text information, the second text information, the first text word, and candidate confusion words. As an example, such as Fig. 9 As shown, the memory 902 may include, but is not limited to, the first acquisition unit 502, the first replacement unit 504, the first recognition unit 506, and the first determination unit 508 in the confusion word mining device of the large language model LLM. In addition, other module units in the confusion word mining device of the large language model LLM may also be included but are not limited to, which will not be repeated in this example.

[0123] Optionally, the transmission device 906 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 906 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 906 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0124] In addition, the electronic device further comprises: a display 908 for displaying the first text information, the second text information, the first text word and candidate confusing words and other information; and a connection bus 910 for connecting various module components in the electronic device.

[0125] In other embodiments, the user device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as a server, user device, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0126] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0127] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0128] It should be noted that the computer system of the electronic device is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0129] The computer system includes a central processing unit (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) or the program loaded from the storage part to the random access memory (RAM). In the random access memory, various programs and data required for system operation are also stored. The central processing unit, the read-only memory and the random access memory are connected to each other through a bus. The input / output interface (I / O interface) is also connected to the bus.

[0130] The following components are connected to the input / output interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a local area network card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the input / output interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0131] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program contains a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions defined in the system of the present application are executed.

[0132] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0133] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0134] S1, obtaining first text information, wherein the first text information includes a first text word;

[0135] S2, when at least one candidate confusing word of the first text word is determined, using the candidate confusing word to replace the first text word in the first text information to obtain second text information;

[0136] S3, inputting the first text information and the second text information into the LLM, and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used for recognizing text information;

[0137] S4: When there is a difference between the first recognition result and the second recognition result, determine that the candidate confusing word and the first text word belong to the same confusing word pair.

[0138] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the electronic device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0139] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0140] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0141] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed user equipment can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0143] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for mining confusing words of a large language model (LLM), characterized in that: include: Acquire first text information, wherein the first text information includes a first text word; When at least one candidate confusing word of the first text word is determined, the first text word in the first text information is replaced with the candidate confusing word to obtain second text information; Inputting the first text information and the second text information into an LLM, and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model trained using multiple samples and used for recognizing text information; When there is a difference between the first recognition result and the second recognition result, it is determined that the candidate confused word and the first text word belong to the same confused word pair.

2. The method according to claim 1, characterized in that After acquiring the first text information, the method further includes: The first text word is input into the LLM, and the LLM is instructed to output a confusing word to obtain the candidate confusing word.

3. The method according to claim 2, characterized in that After inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, the method further includes: When the first recognition result indicates that the first text information has an incorrect meaning, and the second recognition result also indicates that the second text information has an incorrect meaning, inputting the first text word into the LLM, and instructing the LLM to output a confusing word other than the candidate confusing word to obtain a new confusing word; Using the new confusing word to replace the first text word in the first text information to obtain new text information; Input the new text information into the LLM, and instruct the LLM to perform semantic recognition to obtain a third recognition result corresponding to the new text information; When the third recognition result indicates that the new text information has correct meaning, it is determined that the new confusing word and the first text word belong to the same confusing word pair.

4. The method according to claim 1, characterized in that: After determining that the candidate confusing word and the first text word belong to the same confusing word pair, the method further includes: The candidate confusing word and the first text word are added to a confusing word library, wherein the confusing word library is used to store multiple text words with a confusing relationship, and the candidate confusing word and the first text word determined to be the same confusing word pair are considered to have the confusing relationship.

5. The method according to claim 2, characterized in that: After adding the candidate confusing words and the first text word to the confusing word library, the method further includes: Acquire third text information, wherein the third text information includes the second text word; In the case where it is identified that the third text information has an incorrect meaning, determining a third text word having the confusing relationship with the second text word from the confusing word library; The second text word in the third text information is replaced by the third text word to obtain fourth text information.

6. The method according to any one of claims 1 to 5, characterized in that After inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, the method further includes: When the first recognition result indicates that the first text information is correct and the second recognition result indicates that the second text information is incorrect, or when the first recognition result indicates that the first text information is incorrect and the second recognition result indicates that the second text information is correct, determining that there is a difference between the first recognition result and the second recognition result; When the first recognition result indicates that the first text information is correctly expressed, and the second recognition result also indicates that the second text information is correctly expressed, it is determined that the first text information and the second text information belong to a confusing sentence pair.

7. The method according to any one of claims 1 to 5, characterized in that Before inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, the method further includes: obtaining at least one of the following auxiliary recognition information: domain information of the domain to which the first text information belongs, and context information of the information content in which the first text information is located; The step of inputting the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information includes: inputting the auxiliary recognition information, the first text information and the second text information into the LLM and instructing the LLM to perform semantic recognition to obtain the first recognition result and the second recognition result.

8. A device for mining confusing words of a large language model (LLM), characterized in that: include: A first acquiring unit, configured to acquire first text information, wherein the first text information includes a first text word; A first replacement unit is configured to, when at least one candidate confusing word of the first text word is determined, replace the first text word in the first text information with the candidate confusing word to obtain second text information; A first recognition unit is used to input the first text information and the second text information into an LLM, and instruct the LLM to perform semantic recognition to obtain a first recognition result corresponding to the first text information and a second recognition result corresponding to the second text information, wherein the LLM is a neural network model obtained by training with multiple samples and used for recognizing text information; A first determination unit is used to determine that the candidate confusing word and the first text word belong to the same confusing word pair when the first recognition result indicates that the meaning of the first text information is correct and the second recognition result indicates that the meaning of the second text information is incorrect, or when the first recognition result indicates that the meaning of the first text information is incorrect and the second recognition result indicates that the meaning of the second text information is correct.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images