Scene deviation detection method and device of artificial intelligence model and computing equipment

Abstract

The invention discloses a scene deviation detection method and device for an artificial intelligence model and computing equipment. The method comprises the steps that to-be-detected target dialogue information corresponding to the AI model is acquired; determining a first clustering center corresponding to the preset dialogue information and a second clustering center corresponding to the target dialogue information; wherein the preset dialogue information is dialogue information without scene deviation; based on the first clustering center and the second clustering center, determining a first scene deviation degree corresponding to the target dialogue information; wherein the first scene deviation degree is used for indicating the deviation degree between the application scene of the target dialogue information and the application scene of the preset dialogue information; and determining a scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree. According to the invention, the accuracy of scene deviation detection of the AI model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, and computing equipment for detecting scene deviation of an artificial intelligence model. Background Art

[0002] In recent years, the rapid development of artificial intelligence (AI) technology has driven the in-depth application of AI models in a wide range of fields. AI models have demonstrated remarkable capabilities in tasks such as natural language understanding, knowledge reasoning, and dialogue generation, gradually becoming the core technical support for scenarios such as human-computer interaction, intelligent customer service, and professional question-and-answering. For example, in fields such as healthcare, finance, and education, AI models can provide users with timely and accurate information services through semantic parsing and contextual learning, significantly improving information acquisition efficiency.

[0003] However, with the diversification of application scenarios and the dynamic evolution of user needs, user behavior patterns may change with business development, market trends, or external events, leading to deviations in the AI ​​model's application scenarios. Deviations in the AI ​​model's application scenarios can lead to a decrease in the accuracy of the model's output results. Therefore, how to effectively identify deviations in the AI ​​model's application scenarios has become a key issue in improving AI model accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, and computing device for detecting scene deviation of an artificial intelligence model, which can improve the accuracy of detecting scene deviation of an AI model.

[0005] In a first aspect, an embodiment of the present application provides a scene deviation detection method for an artificial intelligence model, the method comprising: obtaining target dialogue information to be detected corresponding to the artificial intelligence AI model; determining a first cluster center corresponding to the preset dialogue information and a second cluster center corresponding to the target dialogue information; wherein the preset dialogue information is dialogue information in which no scene deviation occurs; based on the first cluster center and the second cluster center, determining a first scene deviation degree corresponding to the target dialogue information; wherein the first scene deviation degree is used to indicate the deviation degree between the application scenario of the target dialogue information and the application scenario of the preset dialogue information; based on the first scene deviation degree, determining a scene deviation detection result corresponding to the target dialogue information.

[0006] Based on this implementation, the scene deviation degree of the target dialogue information can be detected by comparing the cluster centers corresponding to the preset dialogue information and the target dialogue information, thereby improving the accuracy of scene deviation detection.

[0007] In one possible implementation, determining the first cluster center corresponding to the preset conversation information and the second cluster center corresponding to the target conversation information includes: performing keyword extraction on the preset conversation information and the target conversation information, respectively, to obtain multiple first keywords in the preset conversation information and multiple second keywords in the target conversation information; clustering the multiple first keywords to obtain the first cluster center corresponding to the preset conversation information; and clustering the multiple second keywords to obtain the second cluster center corresponding to the target conversation information.

[0008] Based on this implementation method, through keyword extraction and clustering of multiple first keywords corresponding to the preset dialogue information and multiple second keywords corresponding to the target dialogue information, the core semantic features of the preset dialogue information and the target dialogue information can be represented based on the cluster center, so as to improve the accuracy of determining the first scene deviation corresponding to the target dialogue information based on the first cluster center and the second cluster center, and further improve the accuracy of determining the scene deviation detection result based on the first scene deviation.

[0009] In another possible implementation, the clustering of the multiple first keywords to obtain the first cluster center corresponding to the preset dialogue information includes: vectorizing each of the multiple first keywords to obtain a first vector corresponding to each of the first keywords; determining the number of identical vectors of each of the first vectors in the multiple first vectors, and determining a first weight corresponding to each of the first vectors based on the number of identical vectors corresponding to each of the first vectors; determining a target grammatical feature corresponding to each of the first vectors in multiple preset grammatical features, and determining a second weight corresponding to each of the first vectors based on the target grammatical feature corresponding to each of the first vectors; wherein the multiple preset grammatical features include nouns, verbs, and adjectives; clustering the multiple first vectors based on the first weight and second weight corresponding to each of the first vectors to obtain the first cluster center.

[0010] Based on this implementation, the first cluster center is calculated based on double weighted clustering of word frequency and part of speech, which can avoid the interference of low-frequency words and non-core words on clustering, thereby making the determined first cluster center more accurate.

[0011] In another possible implementation, the clustering of the multiple second keywords to obtain the second cluster center corresponding to the target dialogue information includes: vectorizing each of the multiple second keywords to obtain a second vector corresponding to each of the second keywords; determining the number of identical vectors of each of the second vectors in the multiple second vectors, and determining a third weight corresponding to each of the second vectors based on the number of identical vectors corresponding to each of the second vectors; determining a target grammatical feature corresponding to each of the second vectors among multiple preset grammatical features, and determining a fourth weight corresponding to each of the second vectors based on the target grammatical feature corresponding to each of the second vectors; wherein the multiple preset grammatical features include nouns, verbs, and adjectives; clustering the multiple second vectors based on the third weight and fourth weight corresponding to each of the second vectors to obtain the second cluster center.

[0012] Based on this implementation, the second cluster center is calculated based on double weighted clustering of word frequency and part of speech, which can avoid the interference of low-frequency words and non-core words on clustering, thereby making the determined second cluster center more accurate.

[0013] In one implementation, determining the first scene deviation corresponding to the target dialogue information based on the first cluster center and the second cluster center includes: determining the similarity between the first cluster center and the second cluster center; and determining the first scene deviation based on the similarity.

[0014] Based on this implementation, the similarity between the first cluster center and the second cluster center can be determined based on multiple distance algorithms, and then the first scene deviation can be determined based on the similarity, thereby improving the accuracy of the first scene deviation.

[0015] In another possible implementation, the above method also includes: determining a preset application scenario corresponding to the above AI model, and determining a second scene deviation corresponding to the above target dialogue information based on the above preset application scenario and the target question information in the above target dialogue information; the above determination of the scene deviation detection result corresponding to the above target dialogue information based on the above first scene deviation includes: determining the above scene deviation detection result corresponding to the above target dialogue information based on the above first scene deviation and the above second scene deviation.

[0016] This implementation not only detects the scene deviation of the target conversation information based on the cluster center, but also compares the preset application scenario corresponding to the AI ​​model with the actual application scenario corresponding to the target question information to detect the scene deviation of the target conversation information. This enables dual detection of scene deviation, further improving the accuracy of scene deviation detection.

[0017] In another possible implementation, the above-mentioned determination of the preset application scenario corresponding to the above-mentioned AI model, and determining the second scenario deviation corresponding to the above-mentioned target dialogue information based on the above-mentioned preset application scenario and the target question information in the above-mentioned target dialogue information, includes: determining the above-mentioned preset application scenario corresponding to the above-mentioned AI model, and determining the actual application scenario corresponding to the above-mentioned target question information; based on the above-mentioned preset application scenario and the above-mentioned actual application scenario, determining the above-mentioned second scenario deviation between the above-mentioned actual application scenario and the above-mentioned preset application scenario.

[0018] Based on this implementation method, the second AI model can determine whether the actual application scenario of the target question information deviates based on the preset application scenario, thereby effectively identifying the scene deviation and quantifying the degree of scene deviation.

[0019] In another possible implementation, the above-mentioned scene deviation detection result corresponding to the above-mentioned target dialogue information is determined based on the above-mentioned first scene deviation and the above-mentioned second scene deviation, including: determining the target scene deviation based on the above-mentioned first scene deviation and the above-mentioned second scene deviation; determining the above-mentioned scene deviation detection result corresponding to the above-mentioned target scene deviation based on the correlation between the preset scene deviation and the preset detection result; wherein the above-mentioned preset scene deviation includes the above-mentioned target scene deviation, and the above-mentioned preset detection result includes the above-mentioned scene deviation detection result.

[0020] Based on this implementation method, the first scene deviation and the second scene deviation can be converted into intuitive detection results, which can facilitate managers to perform targeted optimization of the performance of the AI ​​model based on the detection results.

[0021] In another possible implementation, the above-mentioned obtaining of the target dialogue information to be detected corresponding to the artificial intelligence AI model includes: when the dialogue turn corresponding to the above-mentioned AI model reaches the preset dialogue turn, obtaining the above-mentioned target dialogue information within the above-mentioned preset dialogue turn; or, when the running time of the above-mentioned AI model reaches the preset time, obtaining the above-mentioned target dialogue information corresponding to the above-mentioned AI model within the above-mentioned preset time; or, obtaining candidate dialogue information corresponding to the above-mentioned AI model; wherein the above-mentioned candidate dialogue information includes candidate question information, candidate reply information output by the above-mentioned AI model for the above-mentioned candidate question information, and feedback information corresponding to the above-mentioned candidate reply information; determining whether the above-mentioned candidate reply information includes a rejection reply, and / or, determining whether the above-mentioned candidate reply information is accurate based on the above-mentioned feedback information; if the above-mentioned candidate reply information includes a rejection reply, and / or, determining that the above-mentioned candidate reply information is inaccurate based on the above-mentioned feedback information, then determining the above-mentioned candidate question information and the above-mentioned candidate reply information as the above-mentioned target dialogue information.

[0022] Based on this implementation, by setting the trigger conditions for scene deviation detection of dialogue information, the scene deviation detection process can be automatically triggered when the preset conditions are met (such as the dialogue round corresponding to the AI ​​model reaches the preset dialogue round, or the running time of the AI ​​model reaches the preset time), and the periodicity of scene deviation detection is ensured, so that the scene deviation of the AI ​​model can be discovered in time. Alternatively, when the reply information output by the AI ​​model is not accurate enough (such as the candidate reply information includes a rejection reply, or the feedback information contains negative feedback), the scene deviation detection process can be automatically triggered, so that the scene deviation of the AI ​​model can be discovered in time.

[0023] In the second aspect, an embodiment of the present application also provides a scene deviation detection device for an artificial intelligence model, including: an acquisition module, configured to acquire target dialogue information to be detected corresponding to the artificial intelligence AI model; a first determination module, configured to determine a first cluster center corresponding to the preset dialogue information and a second cluster center corresponding to the above-mentioned target dialogue information; wherein the above-mentioned preset dialogue information is dialogue information in which no scene deviation occurs; a second determination module, configured to determine a first scene deviation degree corresponding to the above-mentioned target dialogue information based on the above-mentioned first cluster center and the above-mentioned second cluster center; wherein the above-mentioned first scene deviation degree is used to indicate the deviation degree between the application scenario of the above-mentioned target dialogue information and the application scenario of the above-mentioned preset dialogue information; a third determination module, configured to determine the scene deviation detection result corresponding to the above-mentioned target dialogue information based on the above-mentioned first scene deviation degree.

[0024] In one possible implementation, the first determination module is specifically configured to: perform keyword extraction on the preset conversation information and the target conversation information respectively to obtain multiple first keywords in the preset conversation information and multiple second keywords in the target conversation information; perform clustering processing on the multiple first keywords to obtain a first cluster center corresponding to the preset conversation information; and perform clustering processing on the multiple second keywords to obtain a second cluster center corresponding to the target conversation information.

[0025] In another possible implementation, the first determination module is specifically configured to: perform vectorization processing on each of the multiple first keywords to obtain a first vector corresponding to each of the first keywords; determine the number of identical vectors of each of the first vectors in the multiple first vectors, and determine the first weight corresponding to each of the first vectors based on the number of identical vectors corresponding to each of the first vectors; determine the target grammatical feature corresponding to each of the first vectors among multiple preset grammatical features, and determine the second weight corresponding to each of the first vectors based on the target grammatical feature corresponding to each of the first vectors; wherein the multiple preset grammatical features include nouns, verbs and adjectives; cluster the multiple first vectors based on the first weight and second weight corresponding to each of the first vectors to obtain the first cluster center.

[0026] In another possible implementation, the first determination module is configured to: perform vectorization processing on each of the multiple second keywords to obtain a second vector corresponding to each of the multiple second keywords; determine the number of identical vectors of each of the multiple second vectors, and determine the third weight corresponding to each of the multiple second vectors based on the number of identical vectors corresponding to each of the multiple second vectors; determine the target grammatical feature corresponding to each of the multiple preset grammatical features, and determine the fourth weight corresponding to each of the multiple second vectors based on the target grammatical feature corresponding to each of the multiple second vectors; wherein the multiple preset grammatical features include nouns, verbs and adjectives; cluster the multiple second vectors based on the third weight and fourth weight corresponding to each of the multiple second vectors to obtain the second cluster center.

[0027] In another possible implementation, the second determining module is specifically configured to: determine the similarity between the first cluster center and the second cluster center; and determine the first scene deviation based on the similarity.

[0028] In another possible implementation, the above-mentioned device also includes: a fourth determination module; the fourth determination module is configured to: determine the preset application scenario corresponding to the above-mentioned AI model, and determine the second scene deviation corresponding to the above-mentioned target dialogue information based on the above-mentioned preset application scenario and the target question information in the above-mentioned target dialogue information; the third determination module is specifically configured to: determine the above-mentioned scene deviation detection result corresponding to the above-mentioned target dialogue information based on the above-mentioned first scene deviation and the above-mentioned second scene deviation.

[0029] In another possible implementation, the fourth determination module is specifically configured to: determine the preset application scenario corresponding to the AI ​​model, and determine the actual application scenario corresponding to the target question information; based on the preset application scenario and the actual application scenario, determine the second scenario deviation between the actual application scenario and the preset application scenario.

[0030] In another possible implementation, the third determination module is specifically configured to: determine the target scene deviation based on the first scene deviation and the second scene deviation; determine the scene deviation detection result corresponding to the target scene deviation based on the correlation between the preset scene deviation and the preset detection result; wherein the preset scene deviation includes the target scene deviation, and the preset detection result includes the scene deviation detection result.

[0031] In another possible implementation, the acquisition module is specifically configured to: when the dialogue turn corresponding to the AI ​​model reaches the preset dialogue turn, acquire the target dialogue information within the preset dialogue turn; or, when the running time of the AI ​​model reaches the preset time, acquire the target dialogue information corresponding to the AI ​​model within the preset time; or, acquire the candidate dialogue information corresponding to the AI ​​model; wherein the candidate dialogue information includes candidate question information, candidate reply information output by the AI ​​model for the candidate question information, and feedback information corresponding to the candidate reply information; determine whether the candidate reply information includes a rejection reply, and / or, based on the feedback information, determine whether the candidate reply information is accurate; if the candidate reply information includes a rejection reply, and / or, based on the feedback information, determine that the candidate reply information is inaccurate, then determine the candidate question information and the candidate reply information as the target dialogue information.

[0032] In a third aspect, an embodiment of the present application further provides a computing device comprising: a processor and a memory; the processor and the memory are coupled; the memory is used to store program instructions; and the processor is used to execute program instructions to perform any method as described in the first aspect above.

[0033] In a fourth aspect, an embodiment of the present application provides a chip, which is used to execute any method as described in the first aspect above.

[0034] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a computer, the method as described in any one of the first aspects is implemented.

[0035] In a sixth aspect, an embodiment of the present application provides a program product, comprising a computer program, which implements any method in the first aspect when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a scene deviation detection method for an artificial intelligence model provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of obtaining target dialogue information to be detected corresponding to an AI model provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a user interface of an AI model provided in an embodiment of the present application;

[0039] Figure 4 A flowchart of an embodiment of the present application for determining a first cluster center corresponding to preset conversation information and a second cluster center corresponding to target conversation information;

[0040] Figure 5 A flowchart of determining a first cluster center corresponding to preset conversation information provided in an embodiment of the present application;

[0041] Figure 6 A flowchart of determining a second cluster center corresponding to target conversation information provided in an embodiment of the present application;

[0042] Figure 7 A flowchart of determining a second scene deviation corresponding to target dialogue information provided in an embodiment of the present application;

[0043] Figure 8 A flowchart of determining a scene deviation detection result corresponding to target dialogue information provided in an embodiment of the present application;

[0044] Figure 9 A flowchart of another method for detecting scene deviation using an artificial intelligence model provided in an embodiment of the present application;

[0045] Figure 10 A schematic diagram of a scene deviation detection device for an artificial intelligence model provided in an embodiment of the present application;

[0046] Figure 11 A schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. To facilitate the clear description of the technical solutions in the embodiments of the present application, the first, second, etc. descriptions in the embodiments of the present application are only used for illustration and to distinguish the described objects. There is no order, nor does it represent a special limitation on the number of devices in the embodiments of the present application, and it does not constitute any limitation on the embodiments of the present application.

[0048] The scene deviation detection method of the artificial intelligence model provided in the embodiment of the present application can be applied to a computing device, which can be a server or a terminal device. The scene deviation detection method of the artificial intelligence model provided in the embodiment of the present application can obtain the dialogue information generated when the user has a dialogue with the AI ​​model or an AI application that integrates multiple AI models, and detect the scene deviation of the dialogue information and obtain the detection result. This enables managers to determine whether the current dialogue information has scene deviation based on the detection results, and the severity of the scene deviation. Among them, the AI ​​model or AI application is applied to exclusive application scenarios, such as medical diagnosis, translation proofreading, image generation, stock analysis, intelligent customer service, online education and other scenarios.

[0049] Figure 1 A flowchart of a scene deviation detection method of an artificial intelligence model provided in an embodiment of the present application is as follows: Figure 1 As shown, the method includes steps 110 to 140.

[0050] Step 110: Obtain target dialogue information to be detected corresponding to the artificial intelligence AI model.

[0051] In some embodiments, during the operation of an AI model, the interaction between the user and the AI ​​model generates corresponding dialogue information, which includes the question information input by the user and the response information output by the AI ​​model to the question information. The user can input the question information through the user interface of the AI ​​model. After the AI ​​model obtains the question information input by the user, it can analyze and process the question information and output the corresponding response information. The question information and response information are text information.

[0052] AI models (or AI applications) have specific application scenarios (i.e., preset application scenarios), which characterize the core functions, application fields, and required technologies (such as text processing, image recognition, etc.) of the AI ​​model. However, application scenario deviation may occur during the dialogue between the AI ​​model and the user. Application scenario deviation may occur because the question information input by the user does not fit the preset application scenario of the AI ​​model, resulting in the AI ​​model generating low-quality responses or directly refusing to answer due to lack of relevant training data; or, it may be that the training data set does not conform to the application scenario, etc., causing the response information output by the AI ​​model to deviate from its own preset application scenario. Therefore, it is possible to detect whether the application scenario of the dialogue information corresponding to the AI ​​model (hereinafter referred to as the target dialogue information) has scenario deviation. Among them, the target dialogue information includes target question information and target response information output by the AI ​​model for the target question information.

[0053] Step 120 : Determine a first cluster center corresponding to the preset conversation information and a second cluster center corresponding to the target conversation information.

[0054] In some embodiments, the preset dialogue information is dialogue information that does not deviate from the scenario, that is, the application scenario of the preset dialogue information is the same as the preset application scenario of the AI ​​model. The preset dialogue information includes preset question information and preset response information corresponding to the preset question information. For example, the preset dialogue information can be obtained from historical dialogue information of the AI ​​model, or it can be compiled by domain experts based on the preset application scenario of the AI ​​model. This embodiment does not limit the method for obtaining the preset dialogue information.

[0055] In some embodiments, after obtaining the preset dialogue information and the target dialogue information, the preset dialogue information and the target dialogue information can be preprocessed respectively, for example, invalid information in the dialogue information (such as garbled characters, blank input, information irrelevant to the model question and answer, etc.) is eliminated, and the question information and reply information in the dialogue information are segmented to obtain formatted dialogue information.

[0056] In some embodiments, after preprocessing the preset conversation information and the target conversation information, cluster centers corresponding to the preset conversation information (hereinafter referred to as first cluster centers) and cluster centers corresponding to the target conversation information (hereinafter referred to as second cluster centers) can be determined, respectively. Cluster centers refer to representative semantic features corresponding to the preset conversation information or the target conversation information, i.e., features that reflect the primary intent of the question or the primary content of the reply. For example, natural language processing (NLP) techniques, such as keyword extraction, vectorization, and clustering, can be used to determine cluster centers.

[0057] In some embodiments, before triggering scene deviation detection for the target conversation information, a first cluster center corresponding to the preset conversation information can be determined and stored in a database. Thus, when performing scene deviation detection on the target conversation information, the preset conversation information corresponding to the same application scenario as the AI ​​model to be detected can be directly determined in the database and the first cluster center corresponding to the preset conversation information can be obtained from the database. This eliminates the need to determine the first cluster center corresponding to the preset conversation information during each scene deviation detection, thereby reducing computational complexity and improving the efficiency of the scene deviation detection process.

[0058] Step 130 : Determine a first scene deviation corresponding to the target dialogue information based on the first cluster center and the second cluster center.

[0059] In some embodiments, after determining the first cluster center corresponding to the preset conversation information and the second cluster center corresponding to the target conversation information, a first scenario deviation corresponding to the target conversation information can be determined based on the similarity between the first cluster center and the second cluster center. The first scenario deviation indicates the scenario deviation between the application scenario of the target conversation information and the application scenario of the preset conversation information. For example, the similarity between the first cluster center and the second cluster center can be determined based on the Euclidean distance, cosine distance, or Wasserstein distance between the first cluster center and the second cluster center.

[0060] Step 140 : Determine a scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree.

[0061] In some embodiments, after determining the first scene deviation, the first scene deviation can be compared with a preset deviation threshold to determine whether the first scene deviation exceeds the preset deviation threshold. If the first scene deviation exceeds the preset deviation threshold, the scene deviation detection result corresponding to the target conversation information can be determined to indicate that the application scene of the target conversation information deviates from the application scene of the preset conversation information. If the first scene deviation does not exceed the preset deviation threshold, the scene deviation detection result corresponding to the target conversation information can be determined to indicate that the application scene of the target conversation information does not deviate from the application scene of the preset conversation information.

[0062] Through the above scheme, the scene deviation of the target dialogue information can be detected by comparing the cluster centers corresponding to the preset dialogue information and the target dialogue information, thereby improving the accuracy and efficiency of scene deviation detection while reducing the detection cost.

[0063] In some embodiments, the above step 110 includes: when the dialogue turn corresponding to the AI ​​model reaches the preset dialogue turn, obtaining the target dialogue information within the preset dialogue turn; or, when the running time of the AI ​​model reaches the preset time, obtaining the target dialogue information corresponding to the AI ​​model within the preset time.

[0064] For example, during the operation of the AI ​​model, the conversation process between the AI ​​model and the user can be monitored in real time to detect whether the current conversation information meets the conditions for triggering the scene deviation detection process. The conditions for triggering the scene deviation detection process for the target conversation information can be that when the conversation round between the AI ​​model and the user reaches a preset conversation round, or the question information output by the AI ​​model reaches a preset number, multiple conversation information generated in the preset conversation round or a preset number of conversation information are obtained as the target conversation information to be detected, and the scene deviation detection process for the target conversation information is triggered. Among them, the conversation round refers to the number of conversations between the user and the AI ​​model. Each round of conversation includes a question information and the reply information output by the AI ​​model for the question information, that is, each round of conversation includes a question and answer pair.

[0065] Exemplarily, or alternatively, the trigger condition may be that when the running time of the AI ​​model reaches a preset time (i.e., after a preset period), the conversation information within the preset time is obtained as the target conversation information to be detected, and the scene deviation detection process for the target conversation information is triggered.

[0066] Through the above scheme, by setting the trigger conditions for scene deviation detection of dialogue information, the scene deviation detection process can be automatically triggered when the preset conditions are met (such as the dialogue turns corresponding to the AI ​​model reach the preset dialogue turns, or the running time of the AI ​​model reaches the preset time), and the periodicity of scene deviation detection is ensured, so that the scene deviation of the AI ​​model can be discovered in time.

[0067] Figure 2 A flowchart of obtaining target dialogue information to be detected corresponding to an AI model is provided in an embodiment of the present application, such as Figure 2 As shown, the above step 110 includes steps 210 to 230.

[0068] Step 210: Obtain candidate dialogue information corresponding to the AI ​​model.

[0069] In some embodiments, the conditions for triggering the scene deviation detection process for the target dialogue information can also be that when the reply information output by the AI ​​model is a rejection reply or the user is dissatisfied with the reply information, the reply information and the question information corresponding to the reply information are obtained as the target dialogue information to be detected, and the scene deviation detection process for the target dialogue information is triggered.

[0070] Exemplarily, the conversation information between the AI ​​model and the user (i.e., candidate conversation information) can be obtained first, wherein the candidate conversation information includes candidate question information, candidate reply information output by the AI ​​model for the candidate question information, and feedback information corresponding to the candidate reply information.

[0071] In some embodiments, after the user inputs a question to the AI ​​model and obtains the reply information output by the AI ​​model, the user can evaluate whether the reply information is accurate or whether it meets the user's own needs. For example, the user interface of the AI ​​model can be provided with a feedback collection function. For example, after the AI ​​model generates a reply information, the user interface can display evaluation options such as "accept (satisfied)" or "reject (dissatisfied)" in the user interface. The user can complete the feedback on the reply information by clicking the corresponding option button.

[0072] In addition, to obtain more detailed user opinions, a corresponding text input box can be set for the reply information, allowing users to enter specific feedback content in the input box, such as feedback on the high accuracy of the reply information or pointing out the shortcomings of the reply information. Alternatively, users can directly enter feedback information in the input box used to enter question information in the user interface, so that the AI ​​model can output corresponding reply information again based on the feedback information.

[0073] refer to Figure 3 A schematic diagram of a user interface of an AI model is shown, Figure 3 As shown, taking the AI ​​model as a medical diagnostic program as an example, a user can enter a question 301 in an input box 31 of the AI ​​model's user interface 30, such as "I've been having a sore throat lately and occasionally have a low-grade fever, which has lasted about three days. What medicine do I need?" After the AI ​​model receives question 301, it analyzes and infers the question using the AI ​​model, obtains a response 302, and then outputs the response 302 in the user interface 30. After viewing the response 302, the user can provide feedback on the response 302 by clicking "Accept" or "Reject." Alternatively, the user can enter feedback on the response 302 in input box 33. Alternatively, the user can enter feedback on the response 302 in input box 31.

[0074] It should be noted that the above-mentioned manner in which the user provides feedback information is only an example and is not limited in this embodiment.

[0075] Step 220 : determining whether the candidate reply information includes a rejection reply, and / or determining whether the candidate reply information is accurate based on the feedback information.

[0076] In some embodiments, after obtaining the candidate dialogue information, it can be determined whether the candidate dialogue information includes a rejection response, where a rejection response refers to a response output by the AI ​​model that cannot provide relevant information or answers to the question information, such as "I can't answer", "I don't have knowledge in this area", "This question is beyond my ability", etc.

[0077] After obtaining the candidate dialogue information, it is also possible to determine whether the feedback information is positive or negative. If the feedback information is positive, the candidate reply information is determined to be inaccurate; if the feedback information is negative, the candidate reply information is determined to be accurate. Positive feedback refers to a positive evaluation of the candidate reply information, such as the user clicking the "Accept" option or the user entering the evaluation of the candidate reply information as "accurate answer" in the input box; negative feedback refers to a negative evaluation of the candidate reply information, such as the user clicking the "Reject" option or the user entering the evaluation of the candidate reply information as "inaccurate answer" or "please answer again" in the input box.

[0078] Exemplarily, it is possible to determine whether the candidate reply information includes a rejection reply based on another AI model (hereinafter referred to as the first AI model) other than the AI ​​model to be tested, and / or determine whether the candidate reply information is accurate based on the feedback information. In some examples, the candidate reply information and / or the feedback information can be input into the first AI model, and a prompt word can be used to instruct the first AI model to determine whether the candidate reply information includes a rejection reply, and / or determine whether the candidate reply information is accurate based on the feedback information. For example, the prompt word can be:

[0079] You are a rater. Please evaluate whether the candidate responses I provided include rejections and whether the feedback is negative. To complete this evaluation, please follow these steps and criteria:

[0080] 1. Identify AI models’ rejection responses. A rejection response refers to a situation where the AI ​​model explicitly states that it cannot provide relevant information or answers, such as “I don’t have knowledge in this area” or “This question is beyond my ability.”

[0081] 2. Identify negative user feedback. Negative feedback refers to situations where users express dissatisfaction with the model's answer or request a re-answer, such as "Your answer is incorrect" or "Please answer again."

[0082] 3. Give a binary score "yes" or "no":

[0083] For the candidate response information: if it is determined that the candidate response information includes a rejection response, the score is "yes"; if it is determined that the candidate response information does not include a rejection response, the score is "no".

[0084] For the feedback information: If it is determined that the feedback information is negative feedback, the score is "yes"; if it is determined that the feedback information is positive feedback, the score is "no".

[0085] Step 230: If the candidate response information includes a rejection response, and / or it is determined based on the feedback information that the candidate response information is inaccurate, then the candidate question information and the candidate response information are determined as the target dialogue information.

[0086] In some embodiments, if the candidate response information includes a rejection response, and / or it is determined based on the feedback information that the candidate response information is inaccurate, that is, the feedback information is negative feedback, then the candidate question information and the candidate response information can be determined as the target dialogue information, and the detection of the scenario deviation of the target dialogue information is triggered.

[0087] Through the above solution, by setting the trigger condition for the detection of the scenario deviation of the dialogue information, the process of the scenario deviation detection can be automatically triggered when the response information output by the AI model is not accurate enough (such as the candidate response information includes a rejection response, or the feedback information includes negative feedback), so that the scenario deviation situation of the AI model can be discovered in time.

[0088] Figure 4 It is a flowchart for determining the first clustering center corresponding to the preset dialogue information and the second clustering center corresponding to the target dialogue information provided by the embodiment of the present application. As Figure 4 shown, the above step 120 includes steps 410 to 430.

[0089] Step 410: Extract keywords from the preset dialogue information and the target dialogue information respectively, and obtain multiple first keywords in the preset dialogue information and multiple second keywords in the target dialogue information.

[0090] In some embodiments, in the process of determining the first clustering center corresponding to the preset dialogue information, keyword extraction can be first performed on the preset dialogue information by using natural language processing technology to determine multiple keywords (i.e., first keywords) in the preset dialogue information.

[0091] Exemplarily, when extracting keywords from the preset dialogue information, the preset dialogue information can be first normalized, such as removing special symbols, stop words (such as "de", "le") and non-text content (such as emoticons, links). Then, a word segmentation tool is used to perform word segmentation on the normalized preset dialogue information, and a joint model of a language representation model (such as a BERT model) and conditional random fields (CRF) can be used to extract keywords from the segmented preset dialogue information, and multiple first keywords are obtained.

[0092] Among them, the BERT model, through its bidirectional Transformer structure, is able to capture the contextual dependencies of words. For example, in the sentence "Diabetic patients should monitor their blood sugar regularly," the semantics of the word "blood sugar" not only depends on "monitoring," but also forms a strong association with "diabetes." The word vectors generated by BERT can accurately represent this global contextual information, avoiding the semantic bias caused by traditional models (such as TF-IDF or Word2Vec) that ignore context. In addition, the same word may have different meanings in different scenarios. Through pre-training and fine-tuning, BERT can dynamically adjust word vectors according to the context and accurately identify the specialized semantics of different fields.

[0093] By modeling the transition probabilities between word labels, CRF can enforce the rationality of label sequences. For example, in the BIO tagging system (B-beginning, I-inside, O-non-entity), CRF can avoid the unreasonable situation where "I-disease" is placed after the "O" label (such as "B-disease I-disease O" is valid, while "B-disease O I-disease" is invalid). In addition, when using BERT alone for label prediction, the dependencies between labels may be ignored due to word-by-word classification. For example, in "Aspirin is used to treat headaches", BERT may mistakenly label "treatment" as "B-drug", while CRF can force the label of "treatment" to be corrected to "B-use" through global optimization, and form a reasonable sequence with "headache (B-disease)". Therefore, the BERT-CRF joint model can improve the accuracy of keyword extraction through context-aware word vector representation and global optimization of label sequences.

[0094] In some embodiments, after obtaining multiple first keywords and multiple second keywords, synonyms in the multiple first keywords and multiple second keywords can be merged to normalize the keywords. For example, a synonym list corresponding to each application field can be constructed in advance based on a word network (Wordnet) and a domain dictionary containing common vocabulary in the application field corresponding to the AI ​​model; then, based on the synonym list, it is determined whether there are words with the same meaning in the multiple first keywords, and whether there are words with the same meaning in the multiple second keywords.

[0095] Step 420 : clustering the multiple first keywords to obtain a first cluster center corresponding to the preset conversation information.

[0096] In some embodiments, after obtaining the multiple first keywords, clustering can be performed on the multiple first keywords to obtain a first cluster center corresponding to the preset conversation information. The first cluster center is a representative keyword among the multiple first keywords representing the preset conversation information, i.e., a keyword that best represents the semantic features of the preset conversation information. For example, clustering the first vectors can be performed using algorithms such as k-means clustering, hierarchical clustering, and density-based spatial clustering of applications with noise (DBSCAN).

[0097] Step 430 : clustering the multiple second keywords to obtain a second cluster center corresponding to the target conversation information.

[0098] In some embodiments, after obtaining the plurality of second keywords, clustering can be performed on the plurality of second keywords to obtain a second cluster center corresponding to the target conversation information. The second cluster center is a representative keyword among the plurality of second keywords representing the target conversation information, i.e., a keyword that best characterizes the semantic features of the target conversation information. The method for clustering the plurality of second keywords can be found in the description of step 420 and will not be further elaborated here.

[0099] Through the above scheme, keywords are extracted and multiple first keywords corresponding to the preset dialogue information and multiple second keywords corresponding to the target dialogue information are clustered respectively, so that the core semantic features of the preset dialogue information and the target dialogue information can be represented based on the cluster center.

[0100] Figure 5 A flowchart of determining a first cluster center corresponding to preset conversation information provided in an embodiment of the present application is as follows: Figure 5 As shown, the above step 420 includes steps 510 to 540.

[0101] Step 510 : performing vectorization processing on each first keyword among the plurality of first keywords to obtain a first vector corresponding to each first keyword.

[0102] In some embodiments, after obtaining the above-mentioned multiple first keywords, the multiple first keywords can be vectorized to map each first keyword to a high-dimensional semantic space and obtain a vector representation (i.e., a first vector) corresponding to each first keyword. Exemplarily, the method for determining the first vector can adopt a BERT model, a weighted average of word vectors, or a sequence encoding, etc., which is not limited in this embodiment.

[0103] Step 520 : Determine the number of identical vectors of each first vector in the plurality of first vectors, and determine a first weight corresponding to each first vector based on the number of identical vectors corresponding to each first vector.

[0104] In some embodiments, after determining multiple first vectors, a corresponding first weight can be assigned to each first vector based on word frequency, thereby improving the accuracy of the first cluster center. For example, the number of occurrences of each first vector in multiple first vectors can be determined, that is, the number of identical vectors of each first vector in multiple first vectors, and the first weight corresponding to each first vector can be determined based on the number of identical vectors corresponding to each first vector and the total number of multiple first vectors. For example, if the first vector corresponding to the first keyword "cold" appears 8 times in multiple first vectors, and the total number of multiple first vectors is 50, then the first weight corresponding to the first vector can be determined to be number of occurrences / total number = 8 / 50 = 0.16.

[0105] Step 530 : Determine a target grammatical feature corresponding to each first vector from a plurality of preset grammatical features, and determine a second weight corresponding to each first vector based on the target grammatical feature corresponding to each first vector.

[0106] In some embodiments, after determining the plurality of first vectors, a corresponding second weight may be assigned to each first vector based on a grammatical feature (i.e., part of speech), thereby further improving the accuracy of the first cluster center. For example, a target grammatical feature corresponding to each first vector may be determined from the plurality of grammatical features, where the plurality of preset grammatical features include nouns, verbs, and adjectives; and then a second weight corresponding to each first vector may be determined based on the target grammatical feature corresponding to each first vector.

[0107] For example, if the grammatical feature of the first vector is a noun, the second weight corresponding to the first vector can be determined to be 1.2; if the grammatical feature of the first vector is a verb, the second weight corresponding to the first vector can be determined to be 1.0; if the grammatical feature of the first vector is an adjective, the second weight corresponding to the first vector can be determined to be 0.8.

[0108] It should be noted that the specific values ​​of the second weights corresponding to the above-mentioned grammatical features are only examples and are not limited in this embodiment.

[0109] Step 540 : performing clustering processing on the plurality of first vectors based on the first weight and the second weight corresponding to each first vector to obtain a first cluster center.

[0110] In some embodiments, after determining the first weight and second weight corresponding to each first vector, a target weight corresponding to each first vector can be determined based on the product of the first weight and the second weight or the sum of the first weight and the second weight. When clustering multiple first vectors, first vectors with higher target weights contribute more to the calculation of the cluster center.

[0111] Exemplarily, in the process of clustering multiple first vectors, some of the first vectors can be randomly determined as multiple initial cluster centers, and the distances from the remaining first vectors (hereinafter referred to as the vectors to be clustered) to each initial cluster center are calculated, and each vector to be clustered and the initial cluster center closest to each vector to be clustered are clustered to obtain clusters corresponding to each initial cluster center. Then, based on the target weight corresponding to the first vector in each cluster, the cluster center in each cluster is recalculated. Next, the distances between each first vector and each cluster center are continued to be calculated, and each first vector and the cluster center closest to each first vector are clustered to obtain clusters corresponding to each cluster center; and the cluster center in each cluster is recalculated. And so on, until the cluster center no longer changes or the preset number of iterations is reached. In some examples, the method for determining the cluster center corresponding to each cluster can refer to formula (1):

[0112]

[0113] Among them, W i is the target weight corresponding to the first vector in each cluster, V i is the first vector in each cluster, and V is the cluster center corresponding to each cluster.

[0114] Taking the medical diagnosis scenario as an example, if the first keywords corresponding to the preset conversation information are "hypertension", "headache", "aspirin", and "blood sugar test", and the first vector (an exemplary two-dimensional simplified vector), first weight (word frequency), second weight (part of speech), and target weight corresponding to each first keyword are shown in Table 1:

[0115] Table 1

[0116] First keyword First vector First weight Second weight Target weight hypertension [0.8,0.2] 0.35 1.2 (noun) 0.42 Headache [0.5,0.7] 0.26 1.2 (noun) 0.312 aspirin [0.3,0.9] 0.17 1.2 (noun) 0.204 Blood glucose testing [0.6,0.4] 0.21 1.0 (verb) 0.21

[0117] You can first select some of the first vectors as the initial cluster centers, such as the first vectors corresponding to "hypertension" and "headache"; then calculate the distances between the first vectors corresponding to "aspirin" and "blood sugar test" and the two initial cluster centers:

[0118]

[0119] Then, the vector to be clustered that is closest to the first vector corresponding to the initial cluster center "hypertension" is the first vector corresponding to "blood sugar test", and the vector to be clustered that is closest to the first vector corresponding to the initial cluster center "headache" is the first vector corresponding to "aspirin". The two initial clusters formed are "hypertension, blood sugar test" (cluster 1) and "headache, aspirin" (cluster 2).

[0120] Next, use formula (1) to calculate the new cluster centers corresponding to the two initial clusters:

[0121] Cluster 1:

[0122] Cluster 2:

[0123] The new cluster center corresponding to cluster 1 is [0.733, 0.267], and the new cluster center corresponding to cluster 2 is [0.421, 0.779].

[0124] Then, the distances between the first vectors corresponding to "hypertension", "headache", "aspirin", and "blood sugar test" and the new cluster centers corresponding to cluster 1 and cluster 2 are calculated to determine the cluster centers again until the cluster centers no longer change or the preset number of iterations is reached.

[0125] Through the above solution, the first cluster center is calculated based on the dual weighted clustering of word frequency and part of speech, which can avoid the interference of low-frequency words and non-core words on the clustering, thereby making the determined first cluster center more accurate.

[0126] Figure 6 A flowchart of determining the second cluster center corresponding to the target conversation information provided in an embodiment of the present application is as follows: Figure 6 As shown, the above step 430 includes steps 610 to 640.

[0127] Step 610 : performing vectorization processing on each second keyword among the plurality of second keywords to obtain a second vector corresponding to each second keyword.

[0128] In some embodiments, after obtaining the plurality of second keywords, vectorization processing can be performed on the plurality of second keywords to map each second keyword to a high-dimensional semantic space and obtain a vector representation (i.e., a first vector) corresponding to each second keyword. Exemplarily, the second vector can be determined using a BERT model, weighted averaging of word vectors, or sequence encoding, etc., which is not limited in this embodiment.

[0129] Step 620 : Determine the number of identical vectors of each second vector in the plurality of second vectors, and determine a third weight corresponding to each second vector based on the number of identical vectors corresponding to each second vector.

[0130] In some embodiments, after determining multiple second vectors, a corresponding third weight can be assigned to each second vector based on word frequency, thereby improving the accuracy of the second cluster center. For example, the number of occurrences of each second vector in the multiple third vectors, i.e., the number of identical vectors for each second vector in the multiple second vectors, can be determined, and the third weight corresponding to each second vector can be determined based on the number of identical vectors corresponding to each second vector and the total number of the multiple second vectors. It will be appreciated that the method for determining the third weight corresponding to each second vector can be referred to the description of step 520 and will not be repeated here.

[0131] Step 630 : Determine a target grammatical feature corresponding to each second vector from a plurality of preset grammatical features, and determine a fourth weight corresponding to each second vector according to the target grammatical feature corresponding to each second vector.

[0132] In some embodiments, after determining multiple second vectors, each second vector may be assigned a corresponding fourth weight based on a grammatical feature (i.e., part of speech), thereby further improving the accuracy of the second cluster center. For example, a target grammatical feature corresponding to each second vector may be determined from the multiple grammatical features; then, based on the target grammatical feature corresponding to each second vector, a fourth weight corresponding to each second vector may be determined. It will be appreciated that the method for determining the fourth weight corresponding to each second vector can be referenced to the description of step 530 and will not be repeated here.

[0133] Step 640 : performing clustering processing on the plurality of second vectors based on the third weight and the fourth weight corresponding to each second vector to obtain a second cluster center.

[0134] In some embodiments, after determining the third and fourth weights corresponding to each second vector, a target weight corresponding to each second vector can be determined based on the product of the third and fourth weights or the sum of the third and fourth weights corresponding to each second vector. During clustering of multiple second vectors, a second vector with a higher target weight contributes more to the calculation of the cluster center. It will be appreciated that the method for determining the second cluster center can be found in the description of step 540 and will not be further elaborated here.

[0135] Through the above solution, the second cluster center is calculated based on the dual weighted clustering of word frequency and part of speech, which can avoid the interference of low-frequency words and non-core words on the clustering, thereby making the determined second cluster center more accurate.

[0136] In some embodiments, step 130 includes: determining the similarity between the first cluster center and the second cluster center; and determining the first scene deviation based on the similarity.

[0137] In some embodiments, after determining the first cluster center corresponding to the preset conversation information and the second cluster center corresponding to the target conversation information, the similarity between the first cluster center and the second cluster center can be determined based on a distance algorithm such as Euclidean distance, cosine distance, or Wasserstein distance. For example, multiple distance algorithms and corresponding weights can be used simultaneously to calculate similarity, such as cosine distance, Wasserstein distance, and corresponding weights.

[0138] Taking the first cluster center V1 as [0.725, 0.275] and the second cluster center V2 as [0.352, 0.420] as an example, the cosine distance CosDis between the first cluster center V1 and the second cluster center V2 is:

[0139]

[0140] The Wasserstein distance W between the first cluster center V1 and the second cluster center V2 is:

[0141]

[0142] If the weight corresponding to the cosine distance is set to 0.6 and the weight corresponding to the Wasserstein distance is set to 0.4, the similarity S between the first cluster center V1 and the second cluster center V2 can be:

[0143] S=0.6×0.1271+0.4×0.4001=0.2363;

[0144] It should be noted that the specific values ​​of the first cluster center and the second cluster center, and the method of determining the similarity between the first cluster center and the second cluster center are merely examples and are not limited in this embodiment.

[0145] In some embodiments, after determining the similarity between the first and second cluster centers, a first scene deviation of the target conversation information relative to the preset conversation information can be determined based on the similarity. For example, the first scene deviation can be calculated as follows: first scene deviation = 1 - similarity S. A larger first scene deviation indicates a more severe scene deviation of the target conversation information. Continuing with the above example, the first scene deviation corresponding to the target conversation information = 1 - 0.2363 = 0.7637.

[0146] Through the above solution, the similarity between the first cluster center and the second cluster center can be determined based on multiple distance algorithms, and then the first scene deviation degree can be determined based on the similarity, thereby improving the accuracy of the first scene deviation degree.

[0147] In some embodiments, the above method also includes: determining a preset application scenario corresponding to the AI ​​model, and determining a second scenario deviation corresponding to the target dialogue information based on the preset application scenario and the target question information in the target dialogue information.

[0148] In some embodiments, in addition to determining the first scenario deviation corresponding to the target dialogue information based on the preset dialogue information, the second scenario deviation corresponding to the target dialogue information can also be determined based on the preset application scenario and target question information corresponding to the AI ​​model. For example, the actual application scenario corresponding to the target question information can be determined by other AI models other than the AI ​​model to be tested, and the second scenario deviation between the actual application scenario corresponding to the target question information and the preset application scenario can be determined. That is, the second scenario deviation is used to indicate the deviation between the actual application scenario of the target question information in the target dialogue information and the preset application scenario of the AI ​​model.

[0149] The preset application scenarios corresponding to the AI ​​model can be determined through the AI ​​model's configuration files, functional documentation, etc. The application scenarios of the AI ​​model may include the AI ​​model's application field (such as medical diagnosis, financial investment, etc.), specific functions (such as providing symptom analysis, medication recommendations, investment recommendations, etc.), and the technologies used (such as text processing, semantic analysis, image recognition, etc.).

[0150] In some embodiments, step 140 includes determining a scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree and the second scene deviation degree.

[0151] In some embodiments, after determining the first scene deviation and the second scene deviation, a target scene deviation can be determined based on the sum of the first scene deviation and the second scene deviation, and the target scene deviation can be compared with a preset deviation threshold to determine whether the target scene deviation exceeds the preset deviation threshold. If the target scene deviation exceeds the preset deviation threshold, the scene deviation detection result corresponding to the target conversation information can be determined as the target conversation information experiencing scene deviation; if the target scene deviation does not exceed the preset deviation threshold, the scene deviation detection result corresponding to the target conversation information can be determined as the target conversation information experiencing no scene deviation.

[0152] Through the above scheme, the scene deviation of the target dialogue information can be double-detected by comparing the cluster centers corresponding to the preset dialogue information and the target dialogue information, and by comparing the preset application scenario corresponding to the AI ​​model and the actual application scenario corresponding to the target question information, thereby further improving the accuracy of scene deviation detection.

[0153] Figure 7 A flowchart of determining the second scene deviation corresponding to the target dialogue information provided in an embodiment of the present application is as follows: Figure 7 As shown, the above-mentioned "determining the preset application scenario corresponding to the AI ​​model, and determining the second scenario deviation corresponding to the target dialogue information based on the preset application scenario and the target question information in the target dialogue information" includes steps 710 to 720.

[0154] Step 710: Determine the preset application scenario corresponding to the AI ​​model and determine the actual application scenario corresponding to the target question information.

[0155] In some embodiments, when determining the deviation degree of the second scenario corresponding to the target dialogue information, the preset application scenario corresponding to the AI ​​model (or a configuration file, functional document, etc. containing the preset usage scenario corresponding to the AI ​​model) and the target question information can be input into other AI models other than the AI ​​model to be tested (hereinafter referred to as the second AI model) to determine the actual application scenario corresponding to the target question information based on the second AI model, and determine whether the actual application scenario deviates from the preset application scenario.

[0156] For example, based on the second AI model, natural language processing technology can be used to determine the application field, implementation function, adopted technology, etc. corresponding to the preset application scenario, and determine the demand field, demand function, demand technology, etc. corresponding to the target question information.

[0157] For example, if the pre-set application scenario for the AI ​​model to be tested is legal consulting, the second AI model can determine that the application field corresponding to the pre-set application scenario is civil and commercial legal consulting. The functions implemented include contract clause parsing, legal clause interpretation, litigation process guidance, and risk assessment. The technologies used include knowledge graph construction, semantic matching, and entity recognition. If the target question information is "Is the liquidated damages agreed in the labor contract legal?", the second AI model can determine that the application field corresponding to the target question information is labor law, the required function is the legality assessment of contract terms, and the required technology is knowledge base matching.

[0158] Step 720 : Based on the preset application scenario and the actual application scenario, determine a second scenario deviation between the actual application scenario and the preset application scenario.

[0159] In some embodiments, the preset application scenario and target question information corresponding to the AI ​​model can be input into the second AI model, and the prompt word can be used to instruct the second AI model to determine the actual application scenario corresponding to the target question information and determine whether the actual application scenario deviates from the preset application scenario. For example, the prompt word can be:

[0160] Please determine the degree of deviation between the actual application scenario corresponding to the target question information and the preset application scenario based on the following conditions:

[0161] 1. Whether the target question information's demand domain falls within the actual application domain of the preset application scenario;

[0162] 2. Whether the required functions of the target question information are within the functional range that can be achieved in the preset application scenario;

[0163] 3. Whether the technology required for the target question information is within the scope of the technology used in the preset application scenario.

[0164] For example, the second AI model can determine the total similarity between the actual application scenario and the preset application scenario by respectively determining the similarity between the demand field and the actual application field, the demand function and the achievable function, and the demand technology and the adopted technology (such as the above similarities can be added and the average value is calculated), and then determine the second scenario deviation based on the total similarity. The second scenario deviation can be: 1-total similarity.

[0165] Through the above scheme, the second AI model can determine whether the actual application scenario of the target question information deviates based on the preset application scenario, thereby effectively identifying the scene deviation and quantifying the degree of scene deviation.

[0166] Figure 8 A flowchart of a method for determining a scene deviation detection result corresponding to target dialogue information provided in an embodiment of the present application is shown as follows: Figure 8 As shown, the above “determining the scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree and the second scene deviation degree” includes steps 810 to 820 .

[0167] Step 810: Determine a target scene deviation based on the first scene deviation and the second scene deviation.

[0168] In some embodiments, after determining the first scene deviation and the second scene deviation corresponding to the target conversation information, a scene deviation detection result corresponding to the target conversation information can be determined based on a preset evaluation rule, the first scene deviation, and the second scene deviation. The preset evaluation rule can include an association between each of a plurality of preset scene deviations and each of a plurality of preset detection results, and the association can be a one-to-one mapping relationship.

[0169] For example, after determining the first scene deviation and the second scene deviation, an average value of the first scene deviation and the second scene deviation may be determined to obtain a target scene deviation.

[0170] Step 820 : Determine a scene deviation detection result corresponding to the target scene deviation based on the correlation between the preset scene deviation and the preset detection result.

[0171] In some embodiments, the preset scene deviation corresponding to the target scene deviation can then be determined according to a preset evaluation rule (i.e., the correlation between the preset detection result and the preset scene deviation), and the target detection result corresponding to the preset scene deviation can be determined as the scene deviation detection result corresponding to the target scene deviation.

[0172] In some examples, the preset evaluation rules may refer to Table 2:

[0173] Table 2

[0174]

[0175]

[0176] Among them, when the deviation from the preset scenario is greater than or equal to 0.9, the preset detection result corresponding to the dialogue information is "completely unrelated to the preset application scenario", that is, the application scenario corresponding to the dialogue information seriously deviates from the preset application scenario of the AI ​​model; when the deviation from the preset scenario is greater than or equal to 0.7 and less than 0.9, the preset detection result corresponding to the dialogue information is "related to the application field, but beyond the functional and technical scope", that is, although the application field corresponding to the dialogue information is related to the application field in the preset application scenario, it exceeds the functional and technical capabilities that the AI ​​model can achieve; when the deviation from the preset scenario is greater than or equal to 0.5 and less than 0.7, the preset detection result corresponding to the dialogue information is "same as the application field, but beyond the functional scope", that is, although the application field corresponding to the dialogue information is the same as the application field in the preset application scenario, it exceeds the functions that the AI ​​model can achieve; when the deviation from the preset scenario is greater than or equal to 0 and less than 0.5, the preset detection result corresponding to the dialogue information is "same as the preset application scenario", that is, the application scenario corresponding to the dialogue information does not deviate from the preset application scenario of the AI ​​model.

[0177] For example, taking the target scene deviation of 0.7637 as an example, according to the preset evaluation rules shown in Table 1, it can be determined that the preset scene deviation corresponding to the target scene deviation is [0.7, 0.9). Therefore, the preset detection result corresponding to the preset scene deviation [0.7, 0.9) "related to the application field, but beyond the functional and technical scope" can be determined as the scene deviation detection result corresponding to the target scene deviation.

[0178] It should be noted that the specific values ​​of the above-mentioned preset scene deviations are only examples and are not limited in this embodiment.

[0179] Through the above solution, the first scenario deviation and the second scenario deviation can be converted into intuitive detection results, which makes it convenient for managers to perform targeted optimization of the performance of the AI ​​model based on the detection results.

[0180] In some embodiments, in the process of monitoring the dialogue information between the AI ​​model and the user and performing scene deviation detection, the scene deviation detection results, the target scene deviation and the corresponding target dialogue information can be stored in a database, and the multiple target dialogue information and the corresponding detection results can be sorted from high to low according to the target scene deviation. Subsequent management personnel can randomly select some detection results from the database, or select the top N detection results with the highest target scene deviation to conduct spot checks on the detection results, so as to determine whether the detection results are accurate through manual review. If it is found through manual review that the proportion of misjudgments of scene deviation (such as judging a situation with a lower scene deviation as a situation with a higher scene deviation) is high, the threshold of the preset scene deviation in the preset evaluation rule can be adjusted, or the accuracy of the AI ​​model used to determine the second scene deviation can be adjusted to reduce the misjudgment of scene deviation.

[0181] In some embodiments, for target conversation information with a high degree of scene deviation, the main keywords in the target conversation information that cause the scene deviation can also be analyzed to determine the dominant factor causing the scene deviation. For example, if there is a target keyword in the target conversation information that appears frequently in the target conversation information but appears less frequently in the preset conversation information, then the target keyword can be determined to be the dominant factor causing the scene deviation. The greater the difference between the frequency of occurrence of the target keyword in the target conversation information and the frequency of occurrence in the preset conversation information, the greater the contribution of the target keyword to the scene deviation. Then, the target keywords can be stored in a database and sorted from high to low according to the degree of difference, so that managers can adjust and optimize the AI ​​model based on the target keywords, such as expanding the knowledge base of the AI ​​model, adjusting the application scenario boundaries, and increasing the training samples of the AI ​​model.

[0182] Figure 9 A flowchart of another scene deviation detection method of an artificial intelligence model provided in an embodiment of the present application is as follows: Figure 9 As shown, the method includes steps 901 to 980.

[0183] Step 910: Monitor the conversation process between the AI ​​model and the user.

[0184] It can be understood that the implementation of step 910 can refer to the description of step 110 and will not be repeated here.

[0185] Step 920: Determine whether the conversation round corresponding to the AI ​​model has reached the preset conversation round, or whether the running time of the AI ​​model has reached the preset time, or whether the candidate reply information of the AI ​​model includes a rejection reply, or whether there is negative feedback in the feedback information.

[0186] Exemplarily, if the conversation turn corresponding to the AI ​​model reaches the preset conversation turn, or the running time of the AI ​​model reaches the preset time, or the candidate reply information of the AI ​​model includes a rejection reply, or there is negative feedback in the feedback information, then step 930 is executed; if the conversation turn corresponding to the I model reaches the preset conversation turn, and the running time of the AI ​​model reaches the preset time, and the candidate reply information of the AI ​​model includes a rejection reply, and there is negative feedback in the feedback information, then step 910 is continued.

[0187] Step 930: If the conversation turn corresponding to the AI ​​model reaches the preset conversation turn, or the running time of the AI ​​model reaches the preset time, or the candidate reply information of the AI ​​model includes a rejection reply, or there is negative feedback in the feedback information, then the target conversation information within the preset conversation turn is obtained.

[0188] It can be understood that the implementation of step 930 can refer to the description of step 110 and step 210 to step 230, and will not be repeated here.

[0189] Step 940 : Clustering the preset dialogue information and the target dialogue information to obtain a first cluster center corresponding to the preset dialogue information and a second cluster center corresponding to the target dialogue information.

[0190] It can be understood that the implementation of step 940 can refer to the description of step 120, step 410 to step 430, step 510 to step 540, and step 610 to step 640, and will not be repeated here.

[0191] Step 950: Determine a first scene deviation corresponding to the target dialogue information based on the first cluster center and the second cluster center.

[0192] It can be understood that the implementation of step 950 can refer to the description of step 130 and will not be repeated here.

[0193] Step 960: Determine the preset application scenario corresponding to the AI ​​model and the actual application scenario corresponding to the target question information.

[0194] It can be understood that the implementation of step 960 can refer to the description of step 710 and will not be repeated here.

[0195] Step 970: Determine a second scenario deviation corresponding to the target dialogue information based on the preset application scenario and the actual application scenario.

[0196] It can be understood that the implementation of step 970 can refer to the description of step 720 and will not be repeated here.

[0197] Step 980: Determine a scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree and the second scene deviation degree.

[0198] It can be understood that the implementation of step 980 can refer to the description of steps 810 to 820, which will not be repeated here.

[0199] By applying the technical solution of the present application, in the process of monitoring the dialogue process of the AI ​​model, by setting the preset conditions for triggering the scene deviation detection, the scene deviation detection can be automatically triggered without relying on manual intervention, thereby improving the real-time and adaptability of the detection. Moreover, by respectively determining the cluster centers of the preset dialogue information and the target dialogue information through keyword extraction and weighted clustering, the cluster center's ability to represent the core semantics can be improved, thereby making the determined first scene deviation more accurate. In addition, by determining the second deviation between the target question information and the preset application scenario, it is possible to determine whether the question information input by the user has a scene deviation. By combining the first scene deviation and the second scene deviation to jointly determine the dual determination method of the scene deviation of the target dialogue information, it is possible to further ensure the accuracy of the scene deviation determination and avoid the occurrence of missed detection, so as to ensure the good operation of the AI ​​model in practical applications.

[0200] Figure 10 Schematic diagram of a scene deviation detection device for an artificial intelligence model provided in an embodiment of the present application. Figure 10 As shown, the scene deviation detection device 1000 of the artificial intelligence model includes an acquisition module 1010, a first determination module 1020, a second determination module 1030 and a third determination module 1040.

[0201] The acquisition module 1010 is configured to obtain the target dialogue information to be detected corresponding to the artificial intelligence AI model.

[0202] The first determining module 1020 is configured to determine a first cluster center corresponding to the preset dialogue information and a second cluster center corresponding to the target dialogue information.

[0203] The preset dialogue information is dialogue information in which no scene deviation occurs.

[0204] The second determining module 1030 is configured to determine a first scene deviation corresponding to the target dialogue information based on the first cluster center and the second cluster center.

[0205] The first scenario deviation is used to indicate the deviation between the application scenario of the target dialogue information and the application scenario of the preset dialogue information.

[0206] The third determining module 1040 is configured to determine a scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree.

[0207] In some embodiments, the first determination module 1020 is specifically configured to: perform keyword extraction on the preset dialogue information and the target dialogue information respectively to obtain multiple first keywords in the preset dialogue information and multiple second keywords in the target dialogue information; perform clustering processing on the multiple first keywords to obtain the first cluster center corresponding to the preset dialogue information; perform clustering processing on the multiple second keywords to obtain the second cluster center corresponding to the target dialogue information.

[0208] In some embodiments, the first determination module 1020 is specifically configured to: perform vectorization processing on each first keyword among multiple first keywords to obtain a first vector corresponding to each first keyword; determine the number of identical vectors of each first vector among multiple first vectors, and determine the first weight corresponding to each first vector based on the number of identical vectors corresponding to each first vector; determine the target grammatical feature corresponding to each first vector among multiple preset grammatical features, and determine the second weight corresponding to each first vector based on the target grammatical feature corresponding to each first vector; wherein the multiple preset grammatical features include nouns, verbs and adjectives; based on the first weight and second weight corresponding to each first vector, cluster the multiple first vectors to obtain a first cluster center.

[0209] In some embodiments, the first determination module 1020 is configured to: perform vectorization processing on each second keyword among multiple second keywords to obtain a second vector corresponding to each second keyword; determine the number of identical vectors of each second vector among multiple second vectors, and determine the third weight corresponding to each second vector based on the number of identical vectors corresponding to each second vector; determine the target grammatical feature corresponding to each second vector among multiple preset grammatical features, and determine the fourth weight corresponding to each second vector based on the target grammatical feature corresponding to each second vector; wherein the multiple preset grammatical features include nouns, verbs and adjectives; cluster the multiple second vectors based on the third weight and fourth weight corresponding to each second vector to obtain a second cluster center.

[0210] In some embodiments, the second determination module 1030 is specifically configured to: determine the similarity between the first cluster center and the second cluster center; and determine the first scene deviation based on the similarity.

[0211] like Figure 10 As shown, the scene deviation detection device 1000 of the artificial intelligence model also includes: a fourth determination module 1050.

[0212] In some embodiments, the fourth determination module 1050 is configured to: determine the preset application scenario corresponding to the AI ​​model, and determine the second scene deviation corresponding to the target dialogue information based on the preset application scenario and the target question information in the target dialogue information; the third determination module 1040 is specifically configured to: determine the scene deviation detection result corresponding to the target dialogue information based on the first scene deviation and the second scene deviation.

[0213] In some embodiments, the fourth determination module 1050 is specifically configured to: determine the preset application scenario corresponding to the AI ​​model, and determine the actual application scenario corresponding to the target question information; based on the preset application scenario and the actual application scenario, determine the second scenario deviation between the actual application scenario and the preset application scenario.

[0214] In some embodiments, the third determination module 1040 is specifically configured to: determine the target scene deviation based on the first scene deviation and the second scene deviation; determine the scene deviation detection result corresponding to the target scene deviation based on the correlation between the preset scene deviation and the preset detection result; wherein the preset scene deviation includes the target scene deviation, and the preset detection result includes the scene deviation detection result.

[0215] In some embodiments, the acquisition module 1010 is specifically configured to: when the dialogue turn corresponding to the AI ​​model reaches the preset dialogue turn, obtain the target dialogue information within the preset dialogue turn; or, when the running time of the AI ​​model reaches the preset time, obtain the target dialogue information corresponding to the AI ​​model within the preset time; or, obtain the candidate dialogue information corresponding to the AI ​​model; wherein the candidate dialogue information includes candidate question information, candidate reply information output by the AI ​​model for the candidate question information, and feedback information corresponding to the candidate reply information; determine whether the candidate reply information includes a rejection reply, and / or, determine whether the candidate reply information is accurate based on the feedback information; if the candidate reply information includes a rejection reply, and / or, determine that the candidate reply information is inaccurate based on the feedback information, then determine the candidate question information and the candidate reply information as the target dialogue information.

[0216] Figure 11 A schematic diagram of a computing device provided for some embodiments of the present application. In some embodiments, the computing device may be a server, terminal device, or the like. The computing device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the scene deviation detection method of the artificial intelligence model described in the above embodiments.

[0217] like Figure 11As shown, the computing device 1100 includes a processor 1101 and a memory 1102 . Exemplarily, the computing device 1100 may further include a communication interface 1103 and a communication bus 1104 .

[0218] The processor 1101, the memory 1102 and the communication interface 1103 communicate with each other via a communication bus 1104. The communication interface 1103 is used to communicate with other devices such as clients or other servers.

[0219] In some embodiments, the processor 1101 is configured to execute a program 1105, specifically, to execute the relevant steps in the above-mentioned embodiment of the method for detecting scene deviation using an artificial intelligence model. Specifically, the program 1105 may include program code, which includes computer-executable instructions.

[0220] For example, the processor 1101 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement some embodiments of the present application. The computing device 1100 may include one or more processors of the same type, such as one or more CPUs, or different types of processors, such as one or more CPUs and one or more ASICs.

[0221] In some embodiments, the memory 1102 is used to store the program 1105. The memory 1102 may include a high-speed RAM memory, and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0222] Program 1105 can be specifically called by processor 1101 to enable computing device 1100 to execute the scene deviation detection method operation of the artificial intelligence model.

[0223] Some embodiments of the present application provide a computer-readable storage medium storing at least one executable instruction. When the executable instruction runs on a computing device 1100, the computing device 1100 executes the scene deviation detection method of the artificial intelligence model in the above-mentioned embodiment.

[0224] The executable instructions can specifically be used to enable the computing device 1100 to execute the scene deviation detection method operation of the artificial intelligence model.

[0225] For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0226] The beneficial effects that can be achieved by the readable storage medium provided in some embodiments of the present application can be referred to the beneficial effects of the scene deviation detection method of the corresponding artificial intelligence model provided above, and will not be repeated here.

[0227] It should be noted that, in the application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0228] Each embodiment in this specification is described in a related manner. Similar parts between the embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For related parts, refer to the description of the method embodiments.

[0229] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0230] For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device.

[0231] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic device, and a portable compact disc read-only memory (CDROM).

[0232] In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in the computer memory. It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof.

[0233] In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0234] The implementation methods described above are only specific implementation methods of the present application and are not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present application should be included in the scope of protection of the present application.

Claims

1. A scene deviation detection method for an artificial intelligence model, characterized in that: include: Obtain the target dialogue information to be detected corresponding to the artificial intelligence AI model; Determining a first cluster center corresponding to preset conversation information and a second cluster center corresponding to the target conversation information; wherein the preset conversation information is conversation information without scene deviation; Determining a first scenario deviation corresponding to the target conversation information based on the first cluster center and the second cluster center; wherein the first scenario deviation is used to indicate a deviation between an application scenario of the target conversation information and an application scenario of the preset conversation information; Based on the first scene deviation degree, a scene deviation detection result corresponding to the target dialogue information is determined.

2. The method according to claim 1, characterized in that The determining of the first cluster center corresponding to the preset conversation information and the second cluster center corresponding to the target conversation information includes: performing keyword extraction on the preset conversation information and the target conversation information respectively to obtain a plurality of first keywords in the preset conversation information and a plurality of second keywords in the target conversation information; performing clustering processing on the plurality of first keywords to obtain a first cluster center corresponding to the preset conversation information; Clustering is performed on the multiple second keywords to obtain a second cluster center corresponding to the target dialogue information.

3. The method according to claim 2, characterized in that The clustering process of the plurality of first keywords to obtain a first cluster center corresponding to the preset conversation information includes: performing vectorization processing on each of the plurality of first keywords to obtain a first vector corresponding to each of the first keywords; Determine the number of identical vectors of each first vector among the plurality of first vectors, and determine a first weight corresponding to each first vector based on the number of identical vectors corresponding to each first vector; Determining a target grammatical feature corresponding to each of the first vectors from a plurality of preset grammatical features, and determining a second weight corresponding to each of the first vectors based on the target grammatical feature corresponding to each of the first vectors; wherein the plurality of preset grammatical features include nouns, verbs, and adjectives; Clustering is performed on the plurality of first vectors based on the first weight and the second weight corresponding to each of the first vectors to obtain the first cluster center.

4. The method according to claim 1, wherein The determining, based on the first cluster center and the second cluster center, a first scene deviation corresponding to the target dialogue information includes: Determining the similarity between the first cluster center and the second cluster center; The first scene deviation is determined based on the similarity.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Determining a preset application scenario corresponding to the AI ​​model, and determining a second scenario deviation corresponding to the target dialogue information based on the preset application scenario and target question information in the target dialogue information; The determining, based on the first scene deviation degree, a scene deviation detection result corresponding to the target dialogue information includes: The scene deviation detection result corresponding to the target dialogue information is determined based on the first scene deviation degree and the second scene deviation degree.

6. The method according to claim 5, characterized in that The determining of a preset application scenario corresponding to the AI ​​model, and determining a second scenario deviation corresponding to the target dialogue information based on the preset application scenario and target question information in the target dialogue information, includes: Determine the preset application scenario corresponding to the AI ​​model, and determine the actual application scenario corresponding to the target question information; Based on the preset application scenario and the actual application scenario, a second scenario deviation between the actual application scenario and the preset application scenario is determined.

7. The method according to claim 5, characterized in that The determining, based on the first scene deviation degree and the second scene deviation degree, the scene deviation detection result corresponding to the target dialogue information includes: determining a target scene deviation based on the first scene deviation and the second scene deviation; Based on the correlation between the preset scene deviation and the preset detection result, the scene deviation detection result corresponding to the target scene deviation is determined; wherein the preset scene deviation includes the target scene deviation, and the preset detection result includes the scene deviation detection result.

8. The method according to any one of claims 1 to 4, characterized in that The step of obtaining target dialogue information to be detected corresponding to the artificial intelligence (AI) model includes: When the conversation round corresponding to the AI ​​model reaches a preset conversation round, obtaining the target conversation information within the preset conversation round; or When the running time of the AI ​​model reaches a preset time, obtaining the target dialogue information corresponding to the AI ​​model within the preset time; or Obtaining candidate dialogue information corresponding to the AI ​​model; wherein the candidate dialogue information includes candidate question information, candidate response information output by the AI ​​model in response to the candidate question information, and feedback information corresponding to the candidate response information; determining whether the candidate reply information includes a rejection reply, and / or determining whether the candidate reply information is accurate based on the feedback information; If the candidate reply information includes a rejection reply, and / or it is determined based on the feedback information that the candidate reply information is inaccurate, the candidate question information and the candidate reply information are determined as the target dialogue information.

9. A scene deviation detection device for an artificial intelligence model, characterized in that: include: An acquisition module is configured to obtain target dialogue information to be detected corresponding to the artificial intelligence AI model; A first determining module is configured to determine a first cluster center corresponding to preset conversation information and a second cluster center corresponding to the target conversation information; wherein the preset conversation information is conversation information without scene deviation; a second determining module configured to determine a first scenario deviation corresponding to the target conversation information based on the first cluster center and the second cluster center; wherein the first scenario deviation is used to indicate a deviation between an application scenario of the target conversation information and an application scenario of the preset conversation information; The third determining module is configured to determine a scene deviation detection result corresponding to the target dialogue information based on the first scene deviation degree.

10. A computing device, characterized in that include: one or more processors; and a memory configured to: store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the scene deviation detection method of the artificial intelligence model according to any one of claims 1-8.

Images