Information generation method and device and electronic equipment

Abstract

The invention provides an information generation method, which comprises the following steps: in response to a confusion identification module, determining that input information contains first content which is easy to confuse, and obtaining auxiliary information containing second content, the first content and the second content being easy to confuse with each other; respectively retrieving the input information and the auxiliary information to obtain a first retrieval result and a second retrieval result; and performing confusion content cleaning processing on the first retrieval result and the second retrieval result to generate output information corresponding to the input information. The invention further provides an information generation device, electronic equipment and a computer readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an information generation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In the knowledge question-answering scenario, Retrieval Augmented Generation (RAG) is a commonly used core technology. After the user enters the question (Query), RAG retrieves knowledge from the knowledge base, and sends the retrieved knowledge to the Large Language Model (LLM) to generate answers and return them to the user. This method of using a large model as an external knowledge base is an effective way to reduce hallucinations and has been widely implemented in vertical fields. Summary of the invention

[0003] The present disclosure provides an information generation method, device, electronic device and computer-readable storage medium.

[0004] One aspect of the present disclosure provides an information generation method, including: in response to an obfuscation identification module determining that input information contains easily confused first content, obtaining auxiliary information containing second content, wherein the first content and the second content are easily confused with each other; searching the input information and the auxiliary information respectively to obtain a first search result and a second search result; performing obfuscation content cleaning processing on the first search result and the second search result to generate output information corresponding to the input information.

[0005] According to an embodiment of the present disclosure, the first content and the second content are easily confused with each other, and the first content and the second content are similar in expression, and the semantics corresponding to the first content and the semantics corresponding to the second content are different.

[0006] According to an embodiment of the present disclosure, input information and auxiliary information are searched respectively to obtain a first search result and a second search result, including: using the input information for fuzzy matching and exact matching to obtain a first search result associated with at least one of the first content and the second content; and using the auxiliary information for exact matching to obtain a second search result associated with the second content.

[0007] According to an embodiment of the present disclosure, the first search result and the second search result are cleaned of obfuscated content to generate output information corresponding to the input information, including: removing the content that overlaps with the second search result from the first search result to obtain a filtered search result; reordering the filtered search result to obtain a target search result; and generating output information corresponding to the input information based on the target search result.

[0008] According to an embodiment of the present disclosure, output information corresponding to input information is generated based on a target retrieval result, including: obtaining prompt information for prompting that the input information contains easily confused first content; or, obtaining prompt information for prompting to distinguish between the first content and the second content; inputting the prompt information and the target retrieval result into a large language model to obtain output information corresponding to the input information.

[0009] According to an embodiment of the present disclosure, the information generation method also includes: using an obfuscation check module to determine whether the output information satisfies the condition that it includes output content corresponding to the first content and does not include output content corresponding to the second content; in response to the obfuscation check module determining that the output information does not meet the condition, returning to the step of separately searching the input information and the auxiliary information until the output information that meets the condition is obtained; in response to the obfuscation check module determining that the output information meets the condition, outputting the output information.

[0010] According to an embodiment of the present disclosure, in response to a confusion identification module determining that input information contains easily confused first content, obtaining auxiliary information containing second content includes: extracting the easily confused first content from the input information; determining second content that is mutually confused with the first content from a database; the database includes multiple easily confused content pairs, and question-answer pairs corresponding to the easily confused content pairs; determining the question-answer pairs corresponding to the second content in the database; and determining the questions in the question-answer pairs corresponding to the second content as auxiliary information.

[0011] According to an embodiment of the present disclosure, the information generation method also includes: using a confusion identification module to classify the input information to obtain a category of the input information, where the category indicates whether the input information contains easily confusing content; wherein, in response to the category indicating that the input information contains easily confusing content, extracting the easily confusing first content from the input information.

[0012] According to an embodiment of the present disclosure, the information generation method also includes: obtaining historical question and answer pairs and question and answer pairs associated with the historical question and answer pairs as a question and answer pair set; for any two question and answer pairs in the question and answer pair set, in response to a question in one of the two question and answer pairs matching an answer in the other question and answer pair, determining the two question and answer pairs as confusing question and answer pairs, and extracting mutually confusing content pairs from the two question and answer pairs; generating a database based on the confusing question and answer pairs and the mutually confusing content pairs.

[0013] Another aspect of the present disclosure also provides an information generating device, including: an obtaining module, used to obtain auxiliary information containing second content in response to the confusion identification module determining that the input information contains easily confused first content, wherein the first content and the second content are easily confused with each other; a retrieval module, used to search the input information and the auxiliary information respectively to obtain a first retrieval result and a second retrieval result; a processing module, used to perform confusion content cleaning processing on the first retrieval result and the second retrieval result to generate output information corresponding to the input information.

[0014] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0015] Another aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the above method.

[0016] Another aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 An exemplary system architecture to which the information generation method and apparatus disclosed herein can be applied is schematically shown;

[0020] Figure 2 The flowchart of the information generation method according to the embodiment of the present disclosure is schematically shown;

[0021] Figure 3 The flowchart of retrieval enhancement generation RAG according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 4 A system structure diagram of an information generation method according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 5 A schematic diagram schematically shows a method for constructing a database according to an embodiment of the present disclosure;

[0024] Figure 6 A flowchart of an information generating method according to another embodiment of the present disclosure is schematically shown;

[0025] Figure 7 The structure block diagram of the information generating device according to the embodiment of the present disclosure is schematically shown;

[0026] Figure 8 A schematic block diagram of an electronic device that can be used to implement the information generating method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Likewise, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of the data involved (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0029] The core modules of RAG are the retrieval module and the generation module. In the knowledge question-answering system in vertical fields, there are many domain-related fine knowledge and concepts. Both the retrieval module and the generation module have a problem: they cannot distinguish confusing knowledge well. For example, in the knowledge question-answering scenario, the user's two queries are:

[0030] “As an administrator, how do I delete certain accounts?”

[0031] “As an administrator, how do I delete files from certain accounts?”

[0032] "Deleted account files" actually refers to deleted files under the account. When a user wants to "delete an account", the search system may retrieve content related to "deleted account files", and vice versa. If LLM cannot distinguish well, it will lead to wrong answers.

[0033] The existing RAG solution does not specifically address this problem. In addition to the RAG solution, if you want to enhance the ability of LLM domain knowledge to distinguish, you will generally fine-tune LLM to inject domain knowledge. However, in actual application scenarios, fine-tuning cannot be updated quickly, and fine-tuning cannot guarantee an effective solution to this problem.

[0034] Figure 1The exemplary system architecture to which the information generation method and device disclosed in the present invention can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0035] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0036] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0037] The information generation method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the information generation device provided in the embodiment of the present disclosure can generally be set in the server 105. The information generation method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the information generation device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0038] Figure 2 The flowchart of the information generating method according to the embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the information generating method 200 includes operations S210 to S230.

[0040] In operation S210, in response to the confusion identification module determining that the input information includes the first content that is easily confused, auxiliary information including the second content is obtained.

[0041] The first content and the second content are easily confused with each other.

[0042] The confusion identification module may be a classifier, which may classify the input information Query, and the classification result indicates whether the input information contains easily confusing content.

[0043] If the confusion identification module identifies that the input information contains easily confused content, the easily confused content, namely the first content, can be extracted from the input information. Based on the easily confused first content, second content that is easily confused with the first content can be obtained.

[0044] The first content and the second content being easily confused with each other may mean that the first content and the second content are similar in expression, but the semantics of the first content and the semantics of the second content are different.

[0045] For example, the first content "delete account" and the second content "delete account files" are similar in expression, but have completely different meanings. "Delete account files" actually refers to deleting files under an account.

[0046] Based on the second content that is easily confused with the first content, auxiliary information including the second content may be determined.

[0047] For example, the input information including the first content is “As an administrator, how do I delete certain accounts?” The auxiliary information including the second content is “As an administrator, how do I delete files of certain accounts?”.

[0048] In operation S220, the input information and the auxiliary information are searched respectively to obtain a first search result and a second search result.

[0049] For example, a retrieval module may be used to retrieve the input information and the auxiliary information respectively.

[0050] In order to obtain comprehensive search results for the input information, the search module can perform fuzzy matching and exact matching on the input information. Exact matching refers to completely hitting the keywords in the input information, such as the first content "delete account", while fuzzy matching refers to similar expressions to the keywords in the input information. Therefore, the search results of fuzzy matching may contain relevant information of the second content "delete account file" that is easily confused with the first content "delete account".

[0051] The search module may perform an exact match on the auxiliary information to obtain search results associated only with the second content.

[0052] In operation S230, the first search result and the second search result are cleaned of obfuscated contents to generate output information corresponding to the input information.

[0053] The confusing contents of the first search result and the second search result refer to the same content detected based on the easily confused first content and the second content. Because the first content and the second content are easily confused, the same content is detected.

[0054] Performing a confusing content cleaning process on the first search result and the second search result may refer to removing the content overlapping with the second search result from the first search result to obtain a filtered search result.

[0055] For example, filters can be used to clean up obfuscated content. The filters can filter out obfuscated search results, thereby improving the accuracy of the search results.

[0056] For example, the filtered search results may be reordered to obtain target search results. The reordering may be performed according to the relevance to the input information, so the target search result may be a search result with a higher relevance to the input information.

[0057] Sending the target retrieval results to the generation module of the RAG system can improve the RAG system's ability to handle confusing content and generate more accurate answers, i.e., output information.

[0058] The generation module of the RAG system can be a large language model LLM, which generates output information based on prompt words and target retrieval results.

[0059] In order to enable the large language model to distinguish easily confused content, confusion knowledge can be explicitly injected into the large language model. For example, the large language model can be prompted to distinguish easily confused content through prompt words.

[0060] According to an embodiment of the present disclosure, prompt information for prompting that the input information contains the first content that is easily confused is obtained; or prompt information for prompting to distinguish the first content from the second content is obtained.

[0061] For example, the prompt message may be “Please note that the input information contains the easily confusing concept of 'delete account'”. The prompt message may also be “'delete account' and 'delete account file' are two different concepts, please distinguish them carefully”.

[0062] The above prompt information and target search results are input into the large language model to obtain output information corresponding to the input information.

[0063] According to an embodiment of the present disclosure, in response to determining that input information contains easily confused first content, auxiliary information containing second content is obtained, the first content and the second content are easily confused with each other, the input information and the auxiliary information are respectively searched to obtain a first search result and a second search result, the first search result and the second search result are processed to clarify the confused content, and the filtered search results obtained are more accurate, the filtered search results are reordered, and the target search results obtained are more relevant to the input information, and the large language model is prompted by prompt words to distinguish easily confused content, which can improve the ability of the RAG system to process confusing content and generate more accurate output information.

[0064] Figure 3 The flowchart of retrieval enhancement and RAG generation according to an embodiment of the present disclosure is schematically shown.

[0065] like Figure 3 As shown, this embodiment is a RAG overall flow chart. This embodiment provides a confusion identification module, which is used to identify whether the user query contains obfuscated content. If it contains obfuscated content, the obfuscated content identification ability enhancement RAG process is followed. If it does not contain obfuscated content, the normal RAG process is followed.

[0066] The above-mentioned confusion identification module can be a classifier. For example, a single classifier is trained for the confusion concepts that are not well handled by the existing RAG. Since it is impossible to collect all common corpora that are not easy to confuse, the collected corpora that are easy to confuse can be used to train a single classifier. The text single classification method widely used for anomaly detection can be used to train the single classifier. For example, a one-class SVM (Support Vector Machine) is used to train a single classification model to identify user queries containing easily confusing content.

[0067] The above normal RAG process is that after the user inputs the query, RAG generates an answer by sending the knowledge retrieved from the knowledge base into the big model and returns it to the user.

[0068] The above-mentioned RAG process with enhanced obfuscated content identification capability is the process of the information generation method provided by the embodiment of the present disclosure.

[0069] The embodiment of the present disclosure adaptively determines whether to go through the RAG process for enhancing the ability to distinguish confusing content based on whether the user's query contains confusing concepts.

[0070] Combine the following Figure 4 The information generation method provided by the embodiment of the present disclosure is further described.

[0071] Figure 4 The system structure diagram of the information generating method according to the embodiment of the present disclosure is schematically shown.

[0072] like Figure 4 As shown, the system of this embodiment includes a retrieval module 410 , a filtering module 420 , a rearrangement module 430 , a large language model LLM 440 , and an obfuscation checking module 450 .

[0073] For example, the user query is input information containing a first content, and the auxiliary query is auxiliary information containing a second content. The first content and the second content are easily confused with each other.

[0074] The user query and the auxiliary query are respectively input into the search module 410 to obtain the first search result and the second search result. Since the first content and the second content are easily confused with each other, the searches based on the first content and the second content may retrieve the same content, which is the confused content.

[0075] The first search result and the second search result are input into the filtering module 420, and the filtering module 420 performs a process of cleaning the obfuscated content of the first search result and the second search result to obtain a filtered search result. For example, the filtering module 420 removes the content that overlaps with the second search result from the first search result to obtain a filtered search result.

[0076] The rearrangement module 430 reorders the filtered search results, for example, based on the correlation between the filtered search results and the user's query, to obtain target search results.

[0077] Prompt information is obtained. The Prompt may be prompt information prompting the user that the Query contains the first content that is easily confused, or may be prompt information prompting the user that the first content and the second content are different concepts.

[0078] The target search result and the prompt information Prompt are input into the large language model LLM 440, and the large language model LLM 440 generates a preliminary answer based on the prompt information and the target search result.

[0079] In order to further improve the accuracy of the answer, the confusion checking module 450 checks the preliminary answer to determine whether the preliminary answer contains concepts related to the first content and does not contain concepts related to the second content.

[0080] For example, the preliminary answer may be identified using a keyword recognition method, with the first content and the second content respectively used as keywords, to determine whether the preliminary answer contains output content related to the first content and does not contain output content related to the second content.

[0081] If the preliminary answer meets the above conditions, the preliminary answer can be output as the final answer.

[0082] If the preliminary answer does not meet the above conditions, the search module 410 can be returned to search again and the subsequent process can be executed until the answer output by the large language model LLM 440 meets the above conditions. The answer meeting the above conditions is output as the final answer and fed back to the user.

[0083] According to the embodiments of the present disclosure, the RAG system with enhanced ability to distinguish obfuscated content performs pre-processing of retrieval filtering, rearrangement, and explicit prompting of obfuscated content before generating answers using the large language model LLM 440. After generating answers using the large language model LLM 440, it performs post-processing of obfuscated content checking, thereby improving the RAG system's ability to perceive, distinguish, and process obfuscated knowledge, thereby improving the accuracy of the output information.

[0084] For the obfuscated content that is not well handled by LLM, the embodiments of the present disclosure mine the easily confused knowledge to form a corpus (ie, a database). The database includes multiple easily confused content pairs and question-answer pairs corresponding to the easily confused content pairs.

[0085] For example, the first content "delete account" and the second content "delete account files" form a confusing content pair. "How to delete an account" and "How to delete account files" form a confusing question-answer pair.

[0086] For another example, the first content "register and log in" and the second content "log in" form an easily confusing content pair. "How to log in" and "How to register and log in" form an easily confusing question-answer pair.

[0087] After the confusion identification module is used to determine that the input information contains the easily confused first content, the first content is extracted from the input information, and the database is used to obtain auxiliary information containing the second content.

[0088] For example, a second content that is easily confused with a first content is determined from a plurality of easily confused content pairs in a database, and a question-answer pair corresponding to the second content is determined from the database, and the question in the question-answer pair is determined as auxiliary information.

[0089] For example, the input information is “how to log in”, the first content is “login”, the second content determined based on the first content is “register and log in”, and the auxiliary information determined based on the second content is “how to register and log in”.

[0090] According to an embodiment of the present disclosure, auxiliary information corresponding to the input information can be determined through a database, and then a RAG process with enhanced obfuscated content identification capability can be executed based on the input information and the auxiliary information.

[0091] The database provided in the embodiments of the present disclosure is described in detail below.

[0092] Figure 5 The diagram schematically shows a method for constructing a database according to an embodiment of the present disclosure.

[0093] like Figure 5As shown, historical question-answer pairs 510 are question-answer pairs that have been generated by LLM, and related question-answer pairs 520 are question-answer pairs that are related or similar to historical question-answer pairs. Historical question-answer pairs 510 and related question-answer pairs 520 form a question-answer pair set 530.

[0094] For example, generating the question-answer pair set 530 includes the following steps.

[0095] Step a: Generate question-answer pairs based on LLM to form set A.

[0096] Step b: based on step a, similar / related question and answer pairs are generated again based on LLM to form set B.

[0097] Step c: Based on step a, similar / related question and answer pairs are extracted from the historical conversation data of the question-answering system to form a set C.

[0098] Although the question-answer pairs in the above set A and set B are currently generated or regenerated, they can be used as historical question-answer pairs after they are generated. The historical conversation data of the question-answering system can be the question-answer pairs generated before the generation of set A and set B. The above set A, set B and set C constitute the question-answer pair set.

[0099] Next, the question-answer pair set 530 may be checked to check the wrongly answered questions and determine which contents are easily confused by the LLM and which are not easily confused by the LLM. For the easily confused questions and answers, the easily confused question-answer pairs and easily confused content pairs are extracted.

[0100] For example, for two question-answer pairs in the question-answer pair set that have similar expressions but different semantics or requirements, if the question in one of the two question-answer pairs matches the answer in the other question-answer pair, the two question-answer pairs can be determined as confusing question-answer pairs, and mutually confusing content pairs can be extracted from the two question-answer pairs. All the extracted confusing question-answer pairs and confusing content pairs are combined into a corpus, i.e., database 540.

[0101] In one example, the confusion identification module used to determine whether the input information contains easily confusing content can be trained using the above database 540. For example, a single classification model is trained using the easily confusing question-answer pair corpus in the database 540, and the single classification model can be used to classify whether the input information contains easily confusing content.

[0102] In an example, the single classification model may output a judgment result of whether the input information contains easily confusing content, and may also output easily confusing content contained in the input content, that is, the first content.

[0103] In another example, the single classification model may only output a judgment result of whether the input information contains easily confusing content. If the judgment result indicates that the input information contains easily confusing content, the easily confusing content in the input information may be determined by matching the input information with easily confusing content pairs in the database.

[0104] Figure 6 The flowchart of the information generating method according to another embodiment of the present disclosure is schematically shown.

[0105] like Figure 6 As shown, this embodiment includes operations S610 to S660.

[0106] In operation S610, auxiliary information that is easily confused with the user Query is found from a database, and first content and second content that are easily confused with each other are extracted.

[0107] For example, after the confusion identification module determines that the user query (Q1) contains easily confusing content, the auxiliary information (Q2) that is easily confused with the user query can be determined from the easily confusing question-answer pairs in the database. The user query and the auxiliary information are easily confused question-answer pairs, which contain easily confusing content pairs. The easily confusing question-answer pairs extracted from the user query and the auxiliary information are mutually confusing first content (C1) and second content (C2).

[0108] It should be noted that the embodiments of the present disclosure do not limit the order of obtaining the auxiliary information and the first content and the second content. For example, the auxiliary information can be first determined from the easily confused question-answer pairs in the database according to the user query, and then the first content and the second content can be extracted from the user query and the auxiliary information. It is also possible to first match the first content contained in the user query from the database according to the user query, then determine the easily confused second content according to the first content, and finally determine the auxiliary information according to the second content.

[0109] In operation S620, the user query and the auxiliary information are respectively input into a retrieval module to obtain a first retrieval result and a second retrieval result.

[0110] For example, Query (Q1) is sent to the retrieval module, and the retrieval module performs exact matching and fuzzy matching on Q1 and the documents in the knowledge base to obtain the document set docsA as the first retrieval result.

[0111] The auxiliary information (Q2) is sent to the retrieval module, and the retrieval module accurately matches Q2 with the documents in the knowledge base to obtain the document set docsB as the second retrieval result.

[0112] In operation S630, the content overlapping with the second search result is removed from the first search result and re-sorted to obtain a target search result.

[0113] For example, remove docsB from docsA and obtain docsA-docsB as the target retrieval result.

[0114] In operation S640, prompt information is obtained, and the prompt information and the target search result are input into the LLM to obtain output information.

[0115] For example, the prompt message may be “Please note that the input information contains the easily confused concept of 'delete account'”, allowing the LLM to identify the concept that is easily confused with “delete account” by itself.

[0116] The prompt message may also be "'Delete account' and 'Delete account file' are two different concepts, please distinguish them carefully", which more explicitly reminds the LLM to distinguish between easily confused concepts.

[0117] In operation S650, it is determined that the output information includes the first content-related information and does not include the second content-related information.

[0118] In operation S660, output information is fed back.

[0119] In order to further improve the accuracy of the LLM output information, the output information may be checked to determine whether the output information contains concepts related to the first content and does not contain concepts related to the second content.

[0120] For example, the output information may be identified using a keyword recognition method, with the first content and the second content respectively used as keywords, to determine whether the output information contains concepts related to the first content and does not contain concepts related to the second content.

[0121] If the initial output information meets the above conditions, the output information can be fed back to the user as the final result.

[0122] If the output information does not meet the above conditions, the process may return to operation S620 and search again until the output information that meets the above conditions is obtained, and the output information that meets the conditions is fed back to the user.

[0123] Figure 7 The structural block diagram of the information generating device according to the embodiment of the present disclosure is schematically shown.

[0124] like Figure 7 As shown, the information generating device 700 of this embodiment includes an obtaining module 710 , a retrieval module 720 and an obfuscation processing module 730 .

[0125] The obtaining module 710 is used for obtaining auxiliary information including second content in response to the confusion identification module determining that the input information includes first content that is easily confused, wherein the first content and the second content are easily confused with each other.

[0126] The search module 720 is used to search the input information and the auxiliary information respectively to obtain a first search result and a second search result.

[0127] The obfuscation processing module 730 is used to perform obfuscated content cleaning processing on the first search result and the second search result to generate output information corresponding to the input information.

[0128] According to an embodiment of the present disclosure, the first content and the second content are easily confused with each other, and the first content and the second content are similar in expression, and the semantics corresponding to the first content and the semantics corresponding to the second content are different.

[0129] According to an embodiment of the present disclosure, the retrieval module 720 includes a first retrieval unit and a second retrieval unit.

[0130] The first retrieval unit is used to perform fuzzy matching and exact matching on the input information to obtain a first retrieval result associated with at least one of the first content and the second content.

[0131] The second retrieval unit is used to accurately match the auxiliary information to obtain a second retrieval result associated with the second content.

[0132] According to an embodiment of the present disclosure, the obfuscation processing module 730 includes a filtering unit, a reordering unit, and a generating unit.

[0133] The filtering unit is used to remove the content overlapping with the second search result from the first search result to obtain a filtered search result.

[0134] The reordering unit is used to reorder the filtered search results to obtain target search results.

[0135] The generating unit is used to generate output information corresponding to the input information based on the target search result.

[0136] According to an embodiment of the present disclosure, the generating unit includes an obtaining subunit and a generating subunit.

[0137] The obtaining subunit is used to obtain prompt information for prompting that the input information contains the first content that is easily confused; or, to obtain prompt information for prompting to distinguish the first content from the second content.

[0138] The generation subunit is used to input the prompt information and the target retrieval result into the large language model to obtain output information corresponding to the input information.

[0139] According to an embodiment of the present disclosure, the information generating device 700 further includes an obfuscation checking module, a judgment module and an output module.

[0140] The obfuscation checking module is used to use the obfuscation checking module to determine whether the output information meets the condition of including output content corresponding to the first content and not including output content corresponding to the second content.

[0141] The judgment module is used for returning to the step of respectively searching the input information and the auxiliary information in response to the obfuscation checking module judging that the output information does not meet the condition, until the output information meeting the condition is obtained.

[0142] The output module is used for outputting the output information in response to the obfuscation checking module determining that the output information satisfies the condition.

[0143] According to an embodiment of the present disclosure, the obtaining module 710 includes an extracting unit, a first determining unit, a second determining unit, and a third determining unit.

[0144] The extraction unit is used to extract the first content that is easily confused from the input information.

[0145] The first determination unit is used to determine second content that is easily confused with the first content from a database; the database includes a plurality of easily confused content pairs and question-answer pairs corresponding to the easily confused content pairs.

[0146] The second determining unit is used to determine a question-answer pair corresponding to the second content in the database.

[0147] The third determination unit is used to determine the questions in the question-answer pair corresponding to the second content as auxiliary information.

[0148] According to an embodiment of the present disclosure, the information generating device 700 further includes a confusion identification module.

[0149] The obfuscation identification module is used to classify the input information using the obfuscation identification module to obtain a category of the input information, where the category indicates whether the input information contains easily obfuscated content.

[0150] The extraction unit is configured to extract the confusing first content from the input information in response to the category indicating that the input information contains confusing content.

[0151] According to an embodiment of the present disclosure, the information generating device 700 further includes a generating module.

[0152] The generation module is used to obtain historical question-answer pairs and question-answer pairs associated with the historical question-answer pairs as a question-answer pair set; for any two question-answer pairs in the question-answer pair set, in response to a question in one of the two question-answer pairs matching an answer in the other question-answer pair, the two question-answer pairs are determined to be confusing question-answer pairs, and mutually confusing content pairs are extracted from the two question-answer pairs; a database is generated based on the confusing question-answer pairs and mutually confusing content pairs.

[0153] Figure 8 A schematic block diagram of an electronic device that can be used to implement the information generating method according to an embodiment of the present disclosure is schematically shown.

[0154] like Figure 8 As shown, the electronic device 800 according to an embodiment of the present disclosure includes a processor 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 to a random access memory (RAM) 803. The processor 801 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 801 may also include an onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0155] In RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, ROM 802, and RAM 803 are connected to each other through a bus 804. The processor 801 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 802 and / or RAM 803. It should be noted that the program can also be stored in one or more memories other than ROM 802 and RAM 803. The processor 801 can also perform the following operations by executing the program stored in the one or more memories: obtaining at least one historical search path for the index map, the index map includes at least two layers, each layer includes at least two nodes, and each historical search path includes at least two nodes in at least two layers; according to at least one historical search path, determining the path information and time information of at least one target node in the index map, wherein the path information of the target node represents the distance from the search start point of at least one historical search path to the target node; based on the path information and time information of the target node, determining the importance of the target node; according to the importance of the target node, adjusting the layer level of the target node in the index map and adjusting the connection relationship between nodes in the same layer.

[0156] According to an embodiment of the present disclosure, the electronic device 800 may further include an input / output (I / O) interface 805, which is also connected to the bus 804. The electronic device 800 may further include one or more of the following components connected to the I / O interface 805: an input portion 806 including a keyboard, a mouse, etc.; an output portion 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 808 including a hard disk, etc.; and a communication portion 809 including a network interface card such as a LAN card, a modem, etc. The communication portion 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage portion 808 as needed.

[0157] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0158] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 802 and / or RAM 803 described above and / or one or more memories other than ROM 802 and RAM 803.

[0159] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0160] The above functions defined in the system / device of the embodiment of the present disclosure are executed when the computer program is executed by the processor 801. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0161] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 809, and / or installed from a removable medium 811. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0162] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the processor 801, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0163] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals. In the technical solution of this disclosure, the user's authorization or consent is obtained before obtaining or collecting user personal information.

[0164] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0165] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in a variety of ways, even if such combinations and / or couplings are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or couplings fall within the scope of the present disclosure.

[0167] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for generating information, comprising: In response to the confusion identification module determining that the input information includes first content that is easily confused, obtaining auxiliary information including second content, wherein the first content and the second content are easily confused with each other; Retrieving the input information and the auxiliary information respectively to obtain a first retrieval result and a second retrieval result; The first search result and the second search result are cleaned of obfuscated contents to generate output information corresponding to the input information.

2. The method according to claim 1, wherein: The first content and the second content are easily confused with each other: the first content and the second content are similar in expression, and the semantics corresponding to the first content are different from the semantics corresponding to the second content.

3. The method according to claim 1, wherein: The input information and the auxiliary information are searched respectively to obtain a first search result and a second search result, including: Performing fuzzy matching and exact matching on the input information to obtain a first search result associated with at least one of the first content and the second content; and The auxiliary information is accurately matched to obtain a second search result associated with the second content.

4. The method according to claim 3, wherein: The performing obfuscated content cleaning processing on the first search result and the second search result to generate output information corresponding to the input information includes: Removing the content overlapping with the second search result from the first search result to obtain a filtered search result; Reordering the filtered search results to obtain target search results; Output information corresponding to the input information is generated based on the target search result.

5. The method according to claim 4, wherein: The generating output information corresponding to the input information based on the target search result includes: Obtaining prompt information for prompting that the input information contains first content that is easily confused; or obtaining prompt information for prompting to distinguish the first content from the second content; The prompt information and the target search result are input into a large language model to obtain output information corresponding to the input information.

6. The method according to claim 1, further comprising: Using an obfuscation checking module to determine whether the output information satisfies a condition that the output information includes output content corresponding to the first content and does not include output content corresponding to the second content; In response to the obfuscation checking module determining that the output information does not satisfy the condition, returning to the step of respectively searching the input information and the auxiliary information until output information satisfying the condition is obtained; In response to the obfuscation checking module determining that the output information satisfies the condition, the output information is output.

7. The method according to claim 1, wherein: The step of obtaining auxiliary information including second content in response to the confusion identification module determining that the input information includes the first content that is easily confused includes: extracting the confusing first content from the input information; Determine from a database a second content that is easily confused with the first content; the database includes a plurality of easily confused content pairs and question-answer pairs corresponding to the easily confused content pairs; Determining a question-answer pair in the database corresponding to the second content; The questions in the question-answer pair corresponding to the second content are determined as the auxiliary information.

8. The method according to claim 7, further comprising: Using the confusion identification module to classify the input information, to obtain a category of the input information, wherein the category indicates whether the input information contains easily confusing content; In which, in response to the category indicating that the input information contains easily confusing content, the easily confusing first content is extracted from the input information.

9. The method according to claim 8, further comprising: Acquire historical question-answer pairs and question-answer pairs associated with the historical question-answer pairs as a question-answer pair set; For any two question-answer pairs in the set of question-answer pairs, in response to a question in one of the two question-answer pairs matching an answer in the other question-answer pair, determining the two question-answer pairs as confusable question-answer pairs, and extracting mutually confusable content pairs from the two question-answer pairs; The database is generated according to the confusable question-answer pairs and the mutually confusable content pairs.

10. An information generating device, comprising: an obtaining module, configured to obtain auxiliary information including second content in response to the confusion identification module determining that the input information includes first content that is easily confused, wherein the first content and the second content are easily confused with each other; A retrieval module, used to search the input information and the auxiliary information respectively to obtain a first retrieval result and a second retrieval result; The obfuscation processing module is used to perform obfuscation content cleaning processing on the first search result and the second search result to generate output information corresponding to the input information.

Images