Abnormality detection method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses an anomaly detection method and device, electronic equipment and a computer readable storage medium. According to the embodiment of the invention, after the question-answer pairs in the knowledge base are obtained, the plurality of connected graphs are determined according to the questions in the question-answer pairs, the connectivity relationship of the question-answer pairs in the connected graphs is determined according to the similarity of the questions, and then the knowledge conflict detection result is determined according to the answers of the question-answer pairs in the same connected graph. And determining whether the answers of the question and answer pairs in the connected graphs have knowledge conflicts or not. According to the embodiment of the invention, the plurality of connected graphs are constructed based on the questions with relatively high similarity, and knowledge conflict detection is realized based on the difference between the answers of the questions of the same connected graphs, so that the efficiency and accuracy of knowledge conflict detection can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly, to an anomaly detection method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] With the continuous development of computer technology, users can obtain the information they want to know through an intelligent question-answering system. The intelligent question-answering system needs to rely on a knowledge base to implement. With the continuous growth of the amount of information and the accelerating frequency of information iteration, the number of question-and-answer pairs stored in the knowledge base is also increasing. Question-and-answer pairs often have time limitations. Therefore, with the continuous update of information, knowledge conflicts (a phenomenon where there are contradictions, inconsistencies, or mutual exclusions between different knowledge contents, theories, methods, etc.) are likely to occur in the knowledge base. Knowledge conflicts will reduce the accuracy of the intelligent question-answering system and thus affect the user experience. Therefore, it is very necessary to perform knowledge conflict detection on the intelligent question-answering system.

[0003] The prior art usually uses manual methods to detect knowledge conflicts in the knowledge base, but this method often has low detection efficiency and low accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an anomaly detection method, apparatus, electronic device, and computer-readable storage medium to construct multiple connected graphs based on questions with high similarity, and implement knowledge conflict detection based on the differences between the answers of each question in the same connected graph. Therefore, the efficiency and accuracy of knowledge conflict detection can be improved.

[0005] In a first aspect, embodiments of the present invention provide an anomaly detection method, and the method includes:

[0006] Obtain each question-and-answer pair in the knowledge base;

[0007] Determine multiple connected graphs according to the questions in each of the question-and-answer pairs, and the connection relationship of each of the question-and-answer pairs in the connected graph is determined according to the similarity of each of the questions;

[0008] Determine a corresponding knowledge conflict detection result according to the answers of each of the question-and-answer pairs in the same connected graph, and the knowledge conflict detection result is used to represent whether there is a knowledge conflict in each of the corresponding answers.

[0009] Optionally, the determining multiple connected graphs according to the questions in each of the question-and-answer pairs includes:

[0010] Determine a matching degree determination result between each of the questions according to the similarity between each of the questions, and the matching degree determination result is used to represent whether a plurality of the corresponding questions are the same question;

[0011] Determine the multiple connected graphs according to each of the matching degree determination results.

[0012] Optionally, the determining the matching degree determination results between each of the questions according to the similarity between each of the questions includes:

[0013] Determine a first question from each of the questions;

[0014] Calculate the similarity between each of the questions and the first question;

[0015] Determine the questions whose similarity to the first question meets a preset similarity condition as second questions;

[0016] Based on a large language model, respectively determine the matching degree determination results between the first question and each of the second questions, and the matching degree determination results are used to characterize whether the first question and the corresponding second question are the same question.

[0017] Optionally, the method further includes:

[0018] Determine a target connected graph, where the target connected graph is a connected graph in which the knowledge conflict detection result indicates that there is a knowledge conflict in each of the answers;

[0019] Obtain the target answer corresponding to the target connected graph;

[0020] Replace the answer corresponding to at least one of the questions in the target connected graph with the target answer.

[0021] Optionally, the method further includes:

[0022] Determine the prompt information for each of the question-and-answer pairs according to each of the question-and-answer pairs and a preset prompt template, where the preset prompt template is determined in an iterative manner according to the accuracy rate corresponding to the abnormal test results of each sample question-and-answer pair in a preset sample set, the abnormal test results are obtained based on a large language model, and the abnormal test results are used to characterize whether the sample question-and-answer pair has an abnormality of a specified type, and the specified type is at least one of off-topic, over-generalization, non-standard expression, and unclear meaning;

[0023] Based on the large language model, generate the abnormal detection results for each of the question-and-answer pairs according to each of the prompt information, and the abnormal detection results are used to characterize whether the question-and-answer pair has the abnormality of the specified type.

[0024] Optionally, the preset prompt template is determined by the following method:

[0025] Obtain an initial prompt template;

[0026] Determine the type tags of each of the sample question-and-answer pairs, where the type tags characterize whether there are anomalies of the specified type in the sample question-and-answer pairs;

[0027] Use the initial prompt template as the intermediate prompt template for the first iteration cycle. In each iteration cycle, determine the intermediate prompt information of each sample question-and-answer pair according to each sample question-and-answer pair and the intermediate prompt template, generate the anomaly test results of each sample question-and-answer pair according to each intermediate prompt information and the large language model, determine the accuracy rate of the intermediate prompt template according to each anomaly detection result and the corresponding type tag, optimize the intermediate prompt template, obtain an optimized prompt template, and update the optimized prompt template of the current iteration cycle as the intermediate prompt template of the next iteration cycle until the accuracy rate meets the preset accuracy rate condition;

[0028] Determine the intermediate prompt template of the last iteration cycle as the preset prompt template.

[0029] In a second aspect, an embodiment of the present invention provides an anomaly detection device, where the device includes:

[0030] A question-and-answer pair acquisition unit, configured to acquire each question-and-answer pair in the knowledge base;

[0031] A connected graph determination unit, configured to determine a plurality of connected graphs according to the questions in each question-and-answer pair, where the connection relationship of each question-and-answer pair in the connected graph is determined according to the similarity of each question;

[0032] A detection result acquisition unit, configured to determine a corresponding knowledge conflict detection result according to the answers of each question-and-answer pair in the same connected graph, where the knowledge conflict detection result is used to characterize whether there is a knowledge conflict in the corresponding answers.

[0033] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory is used to store one or more computer program instructions, and wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of the first aspects.

[0034] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method as described in any one of the first aspects is implemented.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer program product, where the computer program product includes computer programs / instructions, and when the computer programs / instructions are executed by a processor, the method as described in any one of the first aspects is implemented.

[0036] After obtaining each Q&A pair in the knowledge base in the embodiments of the present invention, multiple connected graphs are determined according to the questions in each Q&A pair, where the connection relationship of each Q&A pair in the connected graph is determined according to the similarity of each question. Furthermore, the knowledge conflict detection result is determined according to the answers of each Q&A pair in the same connected graph, so as to determine whether there is a knowledge conflict in the answers of each Q&A pair in each connected graph. The embodiments of the present invention construct multiple connected graphs based on questions with relatively high similarity, and implement knowledge conflict detection based on the differences between the answers of each question in the same connected graph. Therefore, the efficiency and accuracy of knowledge conflict detection can be improved. Description of the Drawings

[0037] Through the following description of the embodiments of the present invention with reference to the drawings, the above and other objects, features and advantages of the present invention will become clearer. In the drawings:

[0038] Figure 1 is a flowchart of the anomaly detection method in the embodiments of the present invention;

[0039] Figure 2 is a flowchart of determining multiple connected graphs in an alternative implementation manner in the embodiments of the present invention;

[0040] Figure 3 is a flowchart of determining the matching degree determination result between each question in an alternative implementation manner in the embodiments of the present invention;

[0041] Figure 4 is a flowchart of the anomaly detection method in the embodiments of the present invention;

[0042] Figure 5 is a flowchart of the anomaly detection method in the embodiments of the present invention;

[0043] Figure 6 is a flowchart of determining a preset prompt template in an alternative implementation manner in the embodiments of the present invention;

[0044] Figure 7 is a data flowchart of the anomaly detection method in the embodiments of the present invention;

[0045] Figure 8 is a schematic diagram of the anomaly detection device in the embodiments of the present invention;

[0046] Figure 9 is a schematic diagram of the electronic device in the embodiments of the present invention. Detailed Embodiments

[0047] The present application will be described based on embodiments, but the present application is not limited to these embodiments. In the following detailed description of the present application, some specific details are described in detail. Those skilled in the art can fully understand the present application without the description of these details. In order to avoid obscuring the essence of the present application, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0048] In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0049] Unless the context clearly requires otherwise, the words "including", "comprising", and the like throughout the application shall be construed in an inclusive sense rather than an exclusive or exhaustive sense; that is, the meaning of "including but not limited to".

[0050] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be construed as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0051] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0052] The intelligent question-answering system needs to rely on a knowledge base. With the continuous growth of the amount of information and the accelerating frequency of information iteration, the number of question-answer pairs stored in the knowledge base is also increasing. Taking the FAQ (Frequently-Asked Questions) knowledge base as an example, the question-answer pairs often have time limitations and cognitive limitations, and are also affected by factors such as knowledge sources and cultural differences. Among them, the time limitation means that the answer in the question-answer pair matches the existing knowledge at the moment when the question is generated and cannot match the knowledge that has not yet been generated; the cognitive limitation means that different people have different cognitions and understandings of the same thing, and this understanding may be one-sided or even inaccurate. Therefore, with the continuous update of information, knowledge conflicts are likely to occur in the knowledge base, that is, the answers corresponding to questions expressing the same meaning are contradictory, inconsistent, or mutually exclusive.

[0053] During the Q&A process, the intelligent Q&A system sometimes randomly selects answers that match the user's question from the FAQ knowledge base as responses. Therefore, there are knowledge conflicts in the FAQ knowledge base, which will reduce the accuracy of the intelligent Q&A system and thus affect the user experience. Therefore, it is very necessary to detect knowledge conflicts in the intelligent Q&A system.

[0054] Existing technologies usually use manual methods to detect knowledge conflicts in the knowledge base. However, the amount of data in the FAQ database is extremely large. Therefore, the detection efficiency of manual methods is low, and the accuracy is not high. Moreover, it is easy to miss detections, misdetect, etc.

[0055] The embodiments of the present invention mainly describe the FAQ knowledge base as an example of the knowledge base. It should be understood that this embodiment is not limited thereto, and existing knowledge bases that can support corresponding functions or knowledge bases that can support corresponding functions with the development of future technologies are within the protection scope of the embodiments of the present invention.

[0056] To solve the above problems, the embodiments of the present invention propose an anomaly detection method, device, electronic device, and computer-readable storage medium to construct multiple connected graphs based on questions with high similarity and implement knowledge conflict detection based on the differences between the answers of questions in the same connected graph. Therefore, the efficiency and accuracy of knowledge conflict detection can be improved.

[0057] Figure 1 is the flowchart of the anomaly detection method of the embodiments of the present invention. As Figure 1 shown, the method of this embodiment includes the following steps:

[0058] Step S100, obtain each Q&A pair in the knowledge base.

[0059] In practical applications, a large number of FAQ Q&A pairs will be stored in the knowledge base. FAQ is usually a document or page that contains questions and corresponding answers frequently asked by users regarding specific topics, specific products, specific services, or specific websites, etc. For example, the FAQ for a specific electronic product may include the basic usage instructions of the electronic product, the download method of application programs, common troubleshooting methods, etc. After existing topics, products, services, or websites are updated, or new topics, products, services, or websites are generated, in order to help users understand the new information, managers will add corresponding FAQ Q&A pairs to the knowledge base.

[0060] Therefore, according to actual needs, this embodiment can obtain each current FAQ Q&A pair in the knowledge base at a predetermined cycle, or after the knowledge base is updated, or obtain each Q&A pair in the knowledge base in other ways. This embodiment does not limit this.

[0061] Step S200: Determine multiple connected graphs according to the questions in each Q&A pair.

[0062] In the same knowledge system, the answers to the same question usually should be highly similar or even the same. Therefore, in order to accurately identify which questions belong to the same question, in this embodiment, multiple connected graphs can be determined according to the similarity between the questions in each Q&A pair, so that the questions in the same connected graph are all used to obtain the same knowledge.

[0063] Figure 2 It is a flowchart for determining multiple connected graphs in an optional implementation manner of the embodiment of the present invention. As Figure 2 shown, in an optional implementation manner of this embodiment, step S200 may include the following steps:

[0064] Step S210: Determine the matching degree judgment result between each question according to the similarity between each question.

[0065] For any multiple questions, the higher the similarity between these questions, the higher the possibility of being the same question. Therefore, in this step, the similarity between each question can be determined respectively, and the matching degree judgment result between each question can be determined according to the similarity between each question, that is, whether these questions are the same question.

[0066] Before calculating the similarity between each question, the questions in each question pair can be converted into vectors by means of word embedding, sentence embedding, bag of words (BOW), one-hot encoding, etc., and the similarity between each question can be calculated according to the vectors of each answer, and then the matching degree judgment result between each question can be determined. This embodiment can adopt various existing similarity calculation methods to calculate the similarity between each vector, such as Euclidean Distance, Cosine Similarity, Pearson Correlation Coefficient, etc., and this embodiment does not limit this.

[0067] Figure 3 It is a flowchart for determining the matching degree judgment result between each question in an optional implementation manner of the embodiment of the present invention. As Figure 3 shown, in an optional implementation manner of this embodiment, step S210 may include the following steps:

[0068] Step S211: Determine the first question from each question.

[0069] In this step, each problem can be respectively determined as the first problem.

[0070] Step S212: Calculate the similarity between each problem and the first problem.

[0071] In this step, any problem can be selected from each problem, and the similarity between this problem and the first problem is calculated. Different similarities have different representational capabilities for the semantic similarity between sentences. For example, the accuracy of the Euclidean distance is easily affected by the dimension of the vector, and the cosine similarity is sensitive to the number of times a word appears in the problem without considering the grammar and semantics of the word. Therefore, optionally, in order to enhance the representational ability of the similarity, this step can use multiple similarity calculation methods to calculate the similarity between each problem and the first problem, and use the weighted sum of the similarities between each problem and the first problem under different similarity calculation methods as the similarity between this problem and the first problem. Specifically, the similarity S between problem i and problem j i,j can be expressed by the following formula:

[0072]

[0073] where is the similarity between problem i and problem j under the k-th similarity calculation method, w k is the weight corresponding to the k-th similarity calculation method, the total number of i and j is N, N is a predetermined integer greater than 1, K is the total number of similarity calculation methods, and K is a predetermined integer greater than 1.

[0074] The weights corresponding to each similarity calculation method can be set according to actual needs, and this embodiment does not limit this. For example, when the vector dimension is low, the weight corresponding to the Euclidean distance can be set to a larger value, such as 0.35, and when the vector dimension is high, the weight corresponding to the Euclidean distance can be set to a smaller value, such as 0.2; when the vector is a sparse vector, that is, most elements in the vector are 0 or close to 0, the weight of the cosine similarity is set to a larger value, and when the vector is a dense vector, that is, most elements in the vector are not 0 or not close to 0, the weight of the cosine similarity is set to a smaller value.

[0075] Optionally, in order to save computing resources, if the similarity between any two problems has been calculated, there is no need to perform repeated calculations. For example: Problem Q1 is "How to reserve a training classroom", and Problem Q2 is "How to reserve a training classroom". When Problem Q1 is determined as the first problem, the similarity between Problem Q1 and Problem Q2 can be calculated, then when Problem Q2 is determined as the first problem, there is no need to calculate the similarity between Problem Q2 and Problem Q1 again.

[0076] Step S213: Determine the problems whose similarity to the first problem meets the preset similarity condition as the second problems.

[0077] After calculating the similarity between the first question and each question in the knowledge base, in this embodiment, the question whose similarity to the first question meets the preset similarity condition can be determined as the second question. The preset similarity condition can be that the similarity to the first question is higher than (or not lower than) the preset similarity threshold, where the preset similarity threshold can be set according to actual needs, and this embodiment does not limit this.

[0078] Step S214: Based on the large language model, determine the matching degree determination results between the first question and each second question respectively.

[0079] Questions with high similarity are more likely to be the same question, but there is also the possibility that they are not the same question. For example, the similarity between "How to improve business English writing ability" and "How to improve business English speaking ability" is very high, but there is a certain difference in their meanings. The former focuses more on written expression ability, and the latter focuses more on immediate expression ability, that is, the ability to convey thoughts, viewpoints, emotions, etc. to others through language and body movements, etc. Therefore, in order to further improve the accuracy of the matching degree determination results, this step can identify the first question and the second question based on the large language model to obtain the matching degree determination results between the first question and each second question.

[0080] Specifically, the prompt information of the large language model can be determined according to the first question, each second question, and the pre-set question template, and this prompt information can be used as the input of the large language model to obtain the corresponding matching degree determination result.

[0081] For example: The pre-set question template is "Determine whether the following two questions are the same question. If they are the same question, output true; if they are not the same question, output false. Question A: {description of question A}, Question B: {description of question B}". The first question is "How to improve business English writing ability", and the second question is "How to improve business English speaking ability". Then, "{description of question A}" can be replaced with "How to improve business English writing ability", and "{description of question B}" can be replaced with "How to improve business English speaking ability" to obtain the corresponding prompt information: Determine whether the following two questions are the same question. If they are the same question, output true; if they are not the same question, output false. Question A: {How to improve business English writing ability}, Question B: {How to improve business English speaking ability}. Then, based on this prompt information, the matching degree determination result between the first question and the second question can be obtained.

[0082] The large language model adopted in this embodiment can be various existing large language models, such as BERT, GPT (Generative Pre-trained Transformer) series models, etc. This embodiment does not limit this.

[0083] Step S220: Determine multiple connected graphs according to each matching degree determination result.

[0084] In this step, according to the matching degree determination results between each question, a connected graph can be established based on multiple questions belonging to the same problem, so as to obtain multiple connected graphs formed by each question in the knowledge base. The method of this embodiment can accurately divide each question-and-answer pair into multiple question-and-answer pair groups expressing different meanings, so the accuracy of knowledge conflict detection can be effectively improved.

[0085] For example, if there is a connection relationship between question A and question B, and there is a connection relationship between question B and question C, then it can be determined that there is a connection relationship between question A and question C, so as to obtain a connected graph including question A, question B, and question C.

[0086] Step S300: Determine the corresponding knowledge conflict detection result according to the answers of each question-and-answer pair in the same connected graph.

[0087] After determining multiple connected graphs, this step can perform knowledge conflict detection on the answers of each question pair in the same connected graph to obtain the corresponding knowledge conflict detection result.

[0088] This step can detect knowledge conflicts in various ways, and this embodiment does not limit this. For example, referring to the formation method of the connected graph of each question, the connected graph of each answer can be formed. If the connected graph of any question is inconsistent with the connected graph of the corresponding answer, it means that there is a knowledge conflict in the answers of each Q&A pair in this connected graph; or, based on the large language model, knowledge conflict detection can be performed on the answers of each Q&A pair in the same connected graph to obtain the corresponding knowledge conflict detection result; or, based on the keyword matching method, it can be determined whether there is a situation where the keywords of the answers of each Q&A pair in the same connected graph are inconsistent. If so, it is determined that the knowledge conflict detection result of this connected graph is that there is a knowledge conflict; or, based on the semantic analysis method, based on a preset semantic analysis model, such as a word vector model (such as GloVe, Global Vectors for Word Representation, a global vector model for single-word representation), a pre-trained language model (such as BERT, Bidirectional Encoder Representations from Transformers, a two-way encoder representation based on transformer), etc., semantic analysis is performed on the answers of each Q&A pair in the same connected graph to determine whether each answer represents the same semantics. If some answers represent different semantics, it is determined that the knowledge conflict detection result of this connected graph is that there is a knowledge conflict.

[0089] Figure 4 is the flowchart of the anomaly detection method according to an embodiment of the present invention. As Figure 4 shown, in an optional implementation manner, the method of this embodiment may further include the following steps:

[0090] Step S400, determine the target connected graph.

[0091] After determining the knowledge conflict detection results corresponding to each connected graph, this step can determine the connected graph whose knowledge conflict result characterizes that there is a knowledge conflict in each answer as the target connected graph to process the answers with knowledge conflicts.

[0092] It is easy to understand that step S400 can be executed after step S300.

[0093] Step S500, obtain the target answer corresponding to the target connected graph.

[0094] In this embodiment, the target answer corresponding to the target connectivity graph can be obtained in various ways. Optionally, each Q&A pair corresponding to the target connectivity graph can be sent to the maintainer of the knowledge base, so that the maintainer of the knowledge base can select one answer from each answer as the target answer according to the current cutting-edge and verified research results, such as referring to the latest published authoritative literature, authoritative papers, academic journals, usage instructions officially released by product manufacturers (or designers), etc., or improve the answer closest to the actual situation to obtain the target answer. Optionally, machine learning can also be used. For example, a natural language processing model can extract the corresponding answer from the current cutting-edge and verified research results as the target answer according to each question in the target connectivity graph. Generally, the shorter the release time and / or the longer the answer length of an answer, the higher the possibility of matching the current cutting-edge and verified research results. Therefore, optionally, the answer with the shortest release time and / or the longest answer length among each answer can be determined as the target answer.

[0095] Step S600: Replace the answer corresponding to at least one question in the target connectivity graph with the target answer.

[0096] After determining the target answer, in order to eliminate knowledge conflicts, this step can replace the answer corresponding to at least one question in the target connectivity graph with the target answer, so as to obtain the updated Q&A pair. Optionally, each question in the target connectivity graph can be merged, only one question is retained, and the answer of the retained question is replaced with the target answer. Optionally, the answers of each question in the target connectivity graph can also be replaced with the target answer.

[0097] Figure 5 is the flowchart of the anomaly detection method according to the embodiment of the present invention. As Figure 5 shown, in an optional implementation manner, the method of this embodiment may further include the following steps:

[0098] Step S700: Determine the prompt information of each Q&A pair according to each Q&A pair and a preset prompt template.

[0099] In practical applications, the question-and-answer pairs in the knowledge base may not only have knowledge conflicts, but also other types of anomalies, such as off-topic answers, over-generalization, non-standard expressions, and unclear meanings. Among them, an off-topic answer means that the answer has a low relevance to the question. For example, when the question is "What are the common varieties of apples?", the answer is "Apples are rich in vitamins and are very good for the body." Over-generalization means that the scope of the question is larger than the scope of the answer, resulting in the answer only covering some cases in the question, or not covering any case in the question. For example, when the question is "What are the reasons for unsatisfactory exam results?", the answer is "Lack of serious study attitude." In fact, unsatisfactory exam results are caused by a combination of multiple factors including study attitude. Non-standard expressions mainly include grammar errors, inappropriate word usage, overly colloquial expressions, misuse of punctuation marks, writing errors (including typos, mixed use of simplified and traditional Chinese characters, etc.). Unclear meaning means that the question is too short and can be understood as having multiple meanings, making it impossible to clarify the specific purpose of the questioner. For example, the question is "Picture".

[0100] Therefore, in order to detect whether the question-and-answer pairs have the above-specified types of anomalies, this step can generate prompt information for each question-and-answer pair based on a preset prompt template to perform anomaly detection on each question-and-answer pair in the knowledge base through a large language model. Optionally, in this embodiment, the question-and-answer pairs newly added to the knowledge base within the current period can be obtained periodically and the corresponding prompt information can be generated, or the existing question-and-answer pairs in the knowledge base can be obtained periodically and the corresponding prompt information can be generated, or the corresponding question-and-answer pairs can be obtained according to actual needs and the corresponding prompt information can be generated. This embodiment does not limit this.

[0101] To ensure the accuracy of anomaly detection, in this embodiment, the preset prompt template is determined iteratively according to the accuracy rate corresponding to the anomaly test results of each sample question-and-answer pair in the preset sample set, where the anomaly test results are used to represent whether the sample question-and-answer pair has the specified type of anomaly. Optionally, the preset sample set can be a set composed of question-and-answer pairs randomly selected from the knowledge base, or can be determined by other means. This embodiment does not limit this.

[0102] Optionally, in this embodiment, other specified types of anomaly detection can be performed after it is determined that each question-and-answer pair has no knowledge conflict, or specified type of anomaly detection can be directly performed, that is, step S700 can be executed simultaneously with step S100, or can be executed after step S100 or step S600. This embodiment does not limit this.

[0103] Figure 6 It is a flowchart for determining a preset prompt template in an optional implementation manner of an embodiment of the present invention. As Figure 6 shown, in an optional implementation manner, this embodiment can determine the preset prompt template through the following steps:

[0104] Step S700A, obtain an initial prompt template.

[0105] In this step, an initial prompt template for instructing the large language model to perform a specified type of anomaly detection can be obtained through various existing methods, such as by manual writing, or the large language model generating according to prompt words, etc. According to the actual settings, the initial prompt template can cover only a specified type of anomaly, or can cover various specified types of anomalies, and this embodiment does not limit this.

[0106] For example, taking the specified type of off-topic answer as an example, the initial prompt template can be: You are a linguistics expert. Please determine whether the content in answer A can answer question Q. If answer A can answer question Q, output true; if answer A cannot answer question Q, output false. The following are several examples for your reference: Q: "What is the reimbursement validity period?" A: "Dear, all expenses must be submitted for reimbursement within three months after the business ends. The time point for submitting reimbursement is subject to the employee's completion of submission in the system. Please submit the reimbursement as soon as possible.", output: true; Q: "What is the address of the flower name system?" A: "If you haven't obtained a flower name during the onboarding process, you can also go to the flower name system to get one after you start working.", output: false; Q: "Emergency claim settlement" A: "[Claim settlement conditions] 1. The designated hospital is the general ward of a local public hospital at or above the second level. 2. The cost of a single medical visit does not exceed 3,000 yuan; [Claim settlement path] On the computer: Please click to submit self-service claim settlement; On the mobile phone: Please click here to submit [Warm reminder] 1. If the cost of a single medical visit (specifically: on the same day in the same department) exceeds 3,000 yuan, please submit the claim settlement materials to Room E, Building D, No. C, Road B, City A, received by Project Team F (021 - 61055588) 2. New employees / newborns can submit claim settlements only after being insured for two months. If the insurance period is less than two months, please keep the claim settlement materials well and submit them after being insured for two months. 3. Please log in to myhr - My benefits - Commercial insurance - Claim settlement guidelines to view the details of the materials required for claim settlement;", output: true; Q: "Can personal information be taken away?" A: "During the employee's tenure in the company, the ownership of all invention achievements obtained due to the position belongs to the company. When an employee leaves the company, only personal files can be taken away. Sensitive data should not be stored on personal mobile storage devices without permission during daily work. If it is necessary for work, a prior filing and approval process must be followed.", output: true; Q: "Computer change file transfer" A: "How to back up the data on your computer. I'm here. To back up the computer data, you can use teamfile or use the DingTalk disk to back up your daily work documents. If the data to be backed up is large, you can also go to the nearest IT service desk and ask the engineer to assist you with the backup and migration.", output: true. The following is the content that you need to judge: Q: "${Q}" A: "${A}", output:.

[0107] Step S700B, determine the type tags of each sample question - answer pair.

[0108] The type label of the sample question-answer pair is used to characterize whether there is an abnormality of a specified type in the sample question-answer pair, and the type label can be determined manually. The setting method of the type label can be set according to actual needs. For example, if the specified type is non-standard expression, if any question-answer pair has an abnormality of non-standard expression, it can be determined that the type label of the question-answer pair under the abnormality type of non-standard expression is yes, otherwise the type label is determined to be no; if the specified type is answering off-topic, if any question-answer pair has an abnormality of answering off-topic, that is, the answer cannot answer the question, it can be determined that the type label of the question-answer pair under the abnormality type of non-standard expression is correct (true), otherwise the type label is determined to be false (false).

[0109] Step S700C: Use the initial prompt template as the intermediate prompt template for the first iteration cycle. In each iteration cycle, determine the intermediate prompt information of each sample question-answer pair according to each sample question-answer pair and the intermediate prompt template, generate the abnormal test results of each sample question-answer pair according to each intermediate prompt information and the large language model, determine the accuracy rate of the intermediate prompt template according to each abnormal detection result and the corresponding type label, optimize the intermediate prompt template, obtain the optimized prompt template, and update the optimized prompt template of the current iteration cycle to the intermediate prompt template of the next iteration cycle until the accuracy rate meets the preset accuracy rate condition.

[0110] In this step, the corresponding intermediate prompt information can be generated multiple times according to each sample question-answer pair and the intermediate prompt template of the current cycle in an iterative manner, and the abnormal test results of each sample question-answer pair can be generated according to the large language model. Furthermore, the accuracy rate of the intermediate prompt template for each iteration cycle is determined according to the ratio of the number of sample question-answer pairs in which the abnormal detection result in the preset sample set matches the type label to the total number of sample question-answer pairs in the preset sample set. If the accuracy rate of any intermediate prompt template meets the preset accuracy rate condition, it means that the current prompt template can better cover the abnormalities existing in each sample question-answer pair. Therefore, the intermediate prompt template does not need to be optimized anymore. Otherwise, the intermediate prompt template still needs to be optimized, so as to update the optimized prompt template of the current iteration cycle to the intermediate prompt template of the next iteration cycle.

[0111] For example, if the initial prompt template is template P1, template P1 can be determined as the intermediate prompt template for the first iteration cycle, and the accuracy rate of template P1 can be determined based on the large language model. If the accuracy rate of template P1 meets the preset accuracy rate condition, then template P1 does not need to be optimized anymore; otherwise, obtain the optimized prompt template of template P1, that is, template P1', and use template P1' as the intermediate prompt template for the second iteration cycle.

[0112] Among them, the preset accuracy rate condition can be that the accuracy rate is higher than (or not lower than) the preset accuracy rate threshold, and the preset accuracy rate threshold can be set according to actual needs, and this embodiment does not limit this.

[0113] Step S700D, determine the intermediate prompt template of the last iteration cycle as the preset prompt template.

[0114] After multiple iterations of optimization, this step can determine the intermediate prompt template of the last iteration cycle as the preset prompt template, and then perform anomaly detection on each Q&A pair in the knowledge base.

[0115] Step S800, based on each prompt message, generate the anomaly detection results of each Q&A pair based on the large language model.

[0116] In this step, each prompt message can be used as the input of the large language model respectively, so as to obtain the anomaly detection results of each Q&A pair.

[0117] Figure 7 is a data flow diagram of the anomaly detection method according to an embodiment of the present invention. Figure 7 The anomaly detection method shown is used to perform anomaly detection of a specified type on Q&A pairs. As Figure 7 shown, step S71 can obtain a preset sample set in which each sample Q&A pair has been type-labeled, step S72 can perform the design and optimization of the prompt template, that is, obtain the initial prompt template and the optimized prompt template as the intermediate prompt template, and step S73 can calculate the accuracy corresponding to the intermediate prompt template and determine whether the accuracy is higher than the preset accuracy threshold; if so, execute step S74 to perform a full-scale detection on each Q&A pair in the knowledge base; if not, return to execute step S72.

[0118] After obtaining each Q&A pair in the knowledge base according to an embodiment of the present invention, a plurality of connected graphs are determined according to the questions in each Q&A pair, wherein the connection relationship of each Q&A pair in the connected graph is determined according to the similarity of each question, and then the knowledge conflict detection result is determined according to the answers of each Q&A pair in the same connected graph, so as to determine whether there is a knowledge conflict in the answers of each Q&A pair in each connected graph. The embodiment of the present invention constructs a plurality of connected graphs based on questions with relatively high similarity, and realizes knowledge conflict detection based on the differences between the answers of questions in the same connected graph, so the efficiency and accuracy of knowledge conflict detection can be improved. Furthermore, based on the preset prompt template and the large language model that have been optimized through multiple rounds of iteration, this embodiment performs anomaly detection on each Q&A pair for at least one type of non-answer, over-generalization, non-standard expression, and unclear meaning, and can accurately improve the accuracy of multiple types of anomaly detection.

[0119] Figure 8 is a schematic diagram of the anomaly detection device according to an embodiment of the present invention. As Figure 8 shown, the anomaly detection device of this embodiment includes a Q&A pair acquisition unit 801, a connected graph determination unit 802, and a detection result acquisition unit 803.

[0120] Among them, the Q&A pair acquisition unit 801 is used to acquire each Q&A pair in the knowledge base; the connected graph determination unit 802 is used to determine a plurality of connected graphs according to the questions in each of the Q&A pairs, and the connection relationship of each Q&A pair in the connected graph is determined according to the similarity of each of the questions; the detection result acquisition unit 803 is used to determine the corresponding knowledge conflict detection result according to the answers of each Q&A pair in the same connected graph, and the knowledge conflict detection result is used to represent whether there is a knowledge conflict in the corresponding answers.

[0121] Further, the connected graph determination unit 802 includes a result acquisition subunit and a connected graph determination subunit.

[0122] Among them, the result acquisition subunit is used to determine the matching degree determination result between each of the questions according to the similarity between each of the questions, and the matching degree determination result is used to represent whether the corresponding plurality of questions are the same question; the connected graph determination subunit is used to determine the plurality of connected graphs according to each of the matching degree determination results.

[0123] Further, the result acquisition subunit includes a first question determination module, a similarity calculation module, a second question determination module, and a result acquisition module.

[0124] Among them, the first question determination module is used to determine a first question from each of the questions; the similarity calculation module is used to calculate the similarity between each of the questions and the first question; the second question determination module is used to determine the questions whose similarity to the first question meets a preset similarity condition as second questions; the result acquisition module is used to respectively determine the matching degree determination result between the first question and each of the second questions based on a large language model, and the matching degree determination result is used to represent whether the first question and the corresponding second question are the same question.

[0125] Further, the device further includes a target graph determination unit, an answer acquisition unit, and an answer replacement unit.

[0126] Among them, the target graph determination unit is used to determine a target connected graph, and the target connected graph is the connected graph in which the knowledge conflict detection result represents that there is a knowledge conflict in each of the answers; the answer acquisition unit is used to acquire the target answer corresponding to the target connected graph; the answer replacement unit is used to replace the answer corresponding to at least one of the questions in the target connected graph with the target answer.

[0127] Further, the device further includes an information determination unit and a result acquisition unit.

[0128] Among them, the information determination unit is used to determine the prompt information of each of the Q&A pairs according to each of the Q&A pairs and a preset prompt template, and the preset prompt template is determined in an iterative manner according to the accuracy rate corresponding to the abnormal test results of each sample Q&A pair in a preset sample set. The abnormal test results are obtained based on a large language model, and the abnormal test results are used to characterize whether the sample Q&A pair has an abnormality of a specified type, and the specified type is at least one of off-topic answers, overgeneralization, non-standard expression, and unclear meaning;

[0129] The result acquisition unit is used to generate the abnormal detection results of each of the Q&A pairs based on the large language model according to each of the prompt information, and the abnormal detection results are used to characterize whether the Q&A pair has the abnormality of the specified type.

[0130] Furthermore, the preset prompt template is determined by a template acquisition unit, a marker determination unit, an iteration unit, and a template determination unit.

[0131] Among them, the template acquisition unit is used to acquire an initial prompt template; the marker determination unit is used to determine the type markers of each of the sample Q&A pairs, and the type markers are used to characterize whether the sample Q&A pair has the abnormality of the specified type; the iteration unit is used to use the initial prompt template as the intermediate prompt template of the first iteration cycle, and in each iteration cycle, determine the intermediate prompt information of each of the sample Q&A pairs according to each of the sample Q&A pairs and the intermediate prompt template, generate the abnormal test results of each of the sample Q&A pairs according to each of the intermediate prompt information and the large language model, determine the accuracy rate of the intermediate prompt template according to each of the abnormal detection results and the corresponding type markers, optimize the intermediate prompt template, obtain an optimized prompt template, and update the optimized prompt template of the current iteration cycle to the intermediate prompt template of the next iteration cycle until the accuracy rate meets a preset accuracy rate condition; the template determination unit is used to determine the intermediate prompt template of the last iteration cycle as the preset prompt template.

[0132] After obtaining each Q&A pair in the knowledge base in the embodiment of the present invention, multiple connected graphs are determined according to the questions in each Q&A pair, and the connection relationship of each Q&A pair in the connected graph is determined according to the similarity of each question. Furthermore, the knowledge conflict detection result is determined according to the answers of each Q&A pair in the same connected graph to determine whether there is a knowledge conflict in the answers of each Q&A pair in each connected graph. The embodiment of the present invention constructs multiple connected graphs based on questions with relatively high similarity and realizes knowledge conflict detection based on the differences between the answers of each question in the same connected graph. Therefore, the efficiency and accuracy of knowledge conflict detection can be improved.

[0133] Figure 9 It is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 9 includes a server, a terminal, etc. AsFigure 9 As shown, the electronic device 9 includes at least one processor 901; and a memory 902 communicatively connected to the at least one processor 901; and a communication component 903 communicatively connected to the scanning device, and the communication component 903 receives and transmits data under the control of the processor 901; wherein, the memory 902 stores instructions executable by the at least one processor 901, and the instructions are executed by the at least one processor 901 to implement the above-mentioned anomaly detection method.

[0134] Specifically, the electronic device includes one or more processors 901 and a memory 902. Figure 9 Taking one processor 901 as an example. The processor 901 and the memory 902 can be connected by a bus or other means. Figure 9 Taking the connection by bus as an example. The memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The processor 901 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in the memory 902, that is, to implement the above-mentioned anomaly detection method.

[0135] The memory 902 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store an option list, etc. In addition, the memory 902 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 902 optionally includes a memory remotely set relative to the processor 901, and these remote memories can be connected to an external device through a network. Examples of the above-mentioned network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] One or more modules are stored in the memory 902, and when executed by one or more processors 901, they implement the anomaly detection method in any of the above method embodiments.

[0137] The above product can execute the method provided in the embodiments of the present application, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference can be made to the method provided in the embodiments of the present application.

[0138] After obtaining each Q&A pair in the knowledge base in the embodiments of the present invention, multiple connected graphs are determined according to the questions in each Q&A pair, wherein the connection relationship of each Q&A pair in the connected graph is determined according to the similarity of each question, and then the knowledge conflict detection result is determined according to the answers of each Q&A pair in the same connected graph, so as to determine whether there is a knowledge conflict in the answers of each Q&A pair in each connected graph. The embodiments of the present invention construct multiple connected graphs based on questions with relatively high similarity, and implement knowledge conflict detection based on the differences between the answers of questions in the same connected graph, so the efficiency and accuracy of knowledge conflict detection can be improved.

[0139] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, and the computer-readable program is used for a computer to execute the above-mentioned partial or all method embodiments.

[0140] That is, those skilled in the art can understand that all or part of the steps of implementing the above-mentioned embodiment methods can be completed by instructing relevant hardware through a program, and this program is stored in a storage medium, including several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in the embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks and other various media that can store program codes.

[0141] The foregoing are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. An anomaly detection method, characterized in that: The method comprises: Obtain each question-answer pair in the knowledge base; Determine a plurality of connected graphs according to the questions in each of the question-answer pairs, wherein the connectivity relationship of each of the question-answer pairs in the connected graph is determined according to the similarity of each of the questions; The corresponding knowledge conflict detection result is determined according to the answer of each question-answer pair in the same connected graph, and the knowledge conflict detection result is used to characterize whether there is a knowledge conflict between the corresponding answers.

2. The method according to claim 1, characterized in that Determining a plurality of connected graphs according to the questions in each of the question-answer pairs comprises: Determine a matching result between the questions according to the similarity between the questions, wherein the matching result is used to indicate whether the corresponding multiple questions are the same question; The plurality of connected graphs are determined according to the matching degree determination results.

3. The method according to claim 2, characterized in that Determining the matching result between the questions according to the similarity between the questions includes: determining a first question from among the questions; Calculating the similarity between each of the questions and the first question; Determine the question whose similarity with the first question satisfies a preset similarity condition as the second question; The matching degree determination results between the first question and each of the second questions are determined based on the large language model, and the matching degree determination results are used to indicate whether the first question and the corresponding second question are the same question.

4. The method according to claim 1, characterized in that: The method further comprises: Determine a target connectivity graph, wherein the target connectivity graph is the connectivity graph in which the knowledge conflict detection result indicates that the answers have knowledge conflicts; Obtaining a target answer corresponding to the target connectivity graph; The answer corresponding to at least one of the questions in the target connectivity graph is replaced with the target answer.

5. The method according to claim 1, characterized in that The method further comprises: Determine prompt information for each question-answer pair according to each question-answer pair and a preset prompt template, wherein the preset prompt template is determined in an iterative manner according to the accuracy rate corresponding to the abnormal test results of each sample question-answer pair in a preset sample set, wherein the abnormal test results are obtained based on a large language model, and the abnormal test results are used to characterize whether the sample question-answer pair has a specified type of abnormality, wherein the specified type is at least one of irrelevant answers, over-generalization, non-standard expressions, and unclear meanings; According to each of the prompt information, an anomaly detection result of each of the question-answer pairs is generated based on a large language model, and the anomaly detection result is used to characterize whether the question-answer pair has the specified type of anomaly.

6. The method according to claim 5, characterized in that The preset prompt template is determined in the following way: Get the initial prompt template; Determine a type tag for each of the sample question-answer pairs, wherein the type tag indicates whether the sample question-answer pair has an anomaly of the specified type; Taking the initial prompt template as the intermediate prompt template of the first iteration cycle, determining the intermediate prompt information of each sample question and answer pair according to each sample question and answer pair and the intermediate prompt template in each iteration cycle, generating the abnormal test result of each sample question and answer pair according to each intermediate prompt information and the large language model, determining the accuracy of the intermediate prompt template according to each abnormal detection result and each corresponding type mark, optimizing the intermediate prompt template to obtain an optimized prompt template, and updating the optimized prompt template of the current iteration cycle to the intermediate prompt template of the next iteration cycle, until the accuracy meets the preset accuracy condition; The intermediate prompt template of the last iteration cycle is determined as the preset prompt template.

7. An abnormality detection device, characterized in that: The device comprises: A question-answer pair acquisition unit, used to acquire each question-answer pair in the knowledge base; A connectivity graph determining unit, configured to determine a plurality of connectivity graphs according to the questions in each of the question-answer pairs, wherein the connectivity relationship between each of the question-answer pairs in the connectivity graph is determined according to the similarity of each of the questions; The detection result acquisition unit is used to determine the corresponding knowledge conflict detection result according to the answers of each question-answer pair in the same connected graph, and the knowledge conflict detection result is used to characterize whether there is a knowledge conflict between the corresponding answers.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program / instructions, which implement the method according to any one of claims 1 to 6 when executed by a processor.

Images