Large model-based agent abnormal input detection method and device and electronic equipment

Abstract

A method, apparatus, and electronic device for detecting abnormal input to an intelligent agent based on a large model are disclosed, relating to the fields of large models, intelligent agents, and artificial intelligence. The method includes: acquiring input content for the intelligent agent; wherein the intelligent agent is associated with a first large model, which outputs model processing results based on the input content; performing semantic understanding on the input content and the service scope description information corresponding to the intelligent agent based on a second large model, and determining the abnormal detection result corresponding to the input content based on the semantic differences between the input content and the service scope description information; wherein the service scope description information corresponding to the intelligent agent is determined based on the intelligent agent's configuration file, which at least describes the definition of the service functions provided by the intelligent agent. This effectively identifies abnormal inputs using semantic transformation techniques such as synonym replacement, sentence reconstruction, and the addition of interfering words, thereby improving the accuracy and reliability of abnormal detection of the intelligent agent's input content.

Description

Technical Field

[0001] This disclosure relates to the fields of large models, intelligent agents, and artificial intelligence technologies. Specifically, it relates to a method, apparatus, and electronic device for detecting abnormal inputs of an intelligent agent based on a large model. Background Technology

[0002] Large-scale intelligent agents have been widely used in fields such as intelligent customer service, automated decision-making, and intelligent assistants. Their core capability is to understand and execute complex tasks through natural language interaction. With the diversification of interaction scenarios, intelligent agents face security threats such as concealment and semantic manipulation.

[0003] In related technologies, detection methods based on keyword matching or static template matching cannot identify abnormal inputs caused by semantic transformation techniques such as synonym replacement, sentence reconstruction, and the addition of interfering words. This may lead to the failure of the detection mechanism, resulting in low accuracy and reliability of anomaly detection. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] Firstly, this disclosure provides a method for detecting abnormal inputs to an intelligent agent based on a large model, the method comprising:

[0006] Obtain input content for the agent; wherein, the agent is associated with a first large model, and the first large model is used to output model processing results based on the input content of the agent;

[0007] Based on the second major model, semantic understanding is performed on the input content and the service scope description information corresponding to the agent. Based on the semantic differences between the input content and the service scope description information, the anomaly detection result corresponding to the input content is determined. The service scope description information corresponding to the agent is determined based on the agent's configuration file, which at least describes the definition of the service functions provided by the agent.

[0008] Secondly, this disclosure provides an anomalous input detection device for intelligent agents based on a large model, the device comprising:

[0009] An acquisition module is used to acquire input content to the agent; wherein, the agent is associated with a first large model, and the first large model is used to output model processing results based on the input content of the agent;

[0010] The determination module is used to perform semantic understanding on the input content and the service scope description information corresponding to the agent based on the second major model, and to determine the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service scope description information; wherein, the service scope description information corresponding to the agent is determined based on the agent's configuration file, and the configuration file is at least used to describe the definition of the service functions provided by the agent.

[0011] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method described in the first aspect.

[0012] Fourthly, this disclosure provides an electronic device, comprising:

[0013] A storage device on which computer programs are stored;

[0014] A processing device for executing the computer program in the storage device to implement the steps of the method in the first aspect.

[0015] Fifthly, this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0016] The above technical solution performs semantic understanding on the input content of the agent and the service scope description information corresponding to the agent based on a large model. Based on the semantic differences between the input content and the service scope description information, it determines the anomaly detection result corresponding to the input content. Using this method, semantic understanding of the agent's service scope description information and input content based on a large model can at least detect anomalies in the input content based on the semantic differences between the input intent and the service functions provided by the agent. This effectively identifies anomalous inputs using semantic transformation techniques such as synonym replacement, sentence reconstruction, and the addition of interfering words, thereby improving the accuracy and reliability of anomaly detection of the agent's input content.

[0017] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0019] Figure 1This is a schematic flowchart illustrating an abnormal input detection method for intelligent agents based on a large model, according to an exemplary embodiment of this disclosure.

[0020] Figure 2 This is a schematic diagram illustrating a method for detecting abnormal input to an intelligent agent according to an exemplary embodiment of the present disclosure;

[0021] Figure 3 This is a schematic diagram illustrating a preset exception rule according to an exemplary embodiment of the present disclosure;

[0022] Figure 4 This is a schematic diagram of the structure of an intelligent agent abnormal input detection device based on a large model, according to an exemplary embodiment of the present disclosure;

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0024] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0025] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0027] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0028] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0030] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0031] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0032] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0033] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0034] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0035] With the diversification of intelligent agent interaction scenarios, the security threats they face are becoming more covert and semantic. For example, by constructing abnormal inputs, intelligent agents can be guided to perform unauthorized operations, leak sensitive data, or generate harmful content, which poses a serious threat to user privacy protection and corporate asset security.

[0036] In related technologies, anomaly detection methods mainly employ the following input detection mechanisms:

[0037] (1) Rule-based keyword matching: String matching is performed using a predefined keyword library (such as malicious instructions, privacy fields), for example, by using regular expressions for matching. This method is effective for explicit exception statements, but cannot recognize semantically equivalent variant expressions such as synonym substitution and syntactic reconstruction.

[0038] (2) Static template matching: A static template library of normal input content is pre-built, and the abnormality of the input content is judged by syntactic structure comparison. This method is applicable to input scenarios with fixed sentence structure, but it will generate a large number of false alarms when processing open domain dialogues, and it is difficult to identify semantic confusion attacks implemented by adding interference words, contextual misleading and other means.

[0039] (3) Single-dimensional intent classification: The input content is classified using a natural language processing model, and the input content is judged to be abnormal by comparing it with the preset categories. The model processing takes a certain amount of time, the response speed is slow, and the detection will fail if the input content is not within the range of the preset categories, or if the intent categories are intersecting or nested.

[0040] Among the aforementioned technologies, detection methods based on keyword matching and static template matching are ineffective in identifying semantic obfuscation attacks, exhibiting weak detection capabilities that make the detection mechanism easily bypassable and security protection ineffective. Detection methods based on keyword matching, static template matching, and fixed intent classification struggle to adapt to dynamic changes in agent business logic, such as function updates and policy adjustments, as well as the continuous evolution of attack methods, easily leading to insufficient detection coverage and high false positive rates.

[0041] In view of this, the present disclosure provides a method, apparatus and electronic device for detecting abnormal inputs of intelligent agents based on large models, so as to solve the above-mentioned technical problems.

[0042] The embodiments of this disclosure will be further explained below with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart illustrating an abnormal input detection method for an intelligent agent based on a large model, according to an exemplary embodiment of this disclosure. (Refer to...) Figure 1 The abnormal input detection method for this intelligent agent may include the following steps:

[0044] S101: Obtain input content for the agent.

[0045] Among them, the agent is associated with the first major model, which is used to output the model processing results based on the input content of the agent.

[0046] For example, an intelligent agent can be an intelligent agent that connects to the platform to achieve capabilities such as intelligent customer service, automated decision-making, and intelligent assistant. Users can interact with the intelligent agent through the intelligent interaction page provided by the platform. Taking the intelligent customer service scenario as an example, the large model associated with the intelligent agent can output dialogue content based on the user's input content, and the input content of the intelligent agent is the user's input content on the dialogue page. The specific settings can be configured according to requirements, and this disclosure does not impose any restrictions on this.

[0047] S102: Based on the second major model, perform semantic understanding on the input content and the service range description information corresponding to the agent, and determine the anomaly detection result corresponding to the input content according to the semantic difference between the input content and the service range description information.

[0048] The service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is used to describe at least the definition of the service functions provided by the intelligent agent.

[0049] It should be noted that the first major model is the large-scale model for agent association, which is determined based on the actual business scenario, while the second major model is the large-scale model used for anomaly detection of the agent's input content. In other words, the first major model and the second major model are different.

[0050] For example, the service scope description information corresponding to the agent is determined based on a configuration file that includes at least the definitions used to describe the service functions provided by the agent. Therefore, by performing semantic understanding on the service scope description information corresponding to the agent through the large model, at least the actual service functions provided by the agent can be obtained. By performing semantic understanding on the input content through the large model, the expected service functions of the agent can be determined. By comparing whether the actual service functions match the expected service functions, it can be determined whether the input content is abnormal. For example, if the large model associated with the agent is a large model for querying the weather, and the input content is "query order xx", the actual service functions provided by the large model do not match the expected service functions of the agent, then the input content is abnormal.

[0051] By employing the above method, semantic understanding of the service scope description information and input content of the agent is performed based on a large model. At least based on the semantic differences between the input intent of the input content and the service functions provided by the agent, anomaly detection of the input content can be performed. This can effectively identify abnormal inputs caused by semantic transformation methods such as synonym replacement, sentence reconstruction, and the addition of interference words, thereby improving the accuracy and reliability of anomaly detection of the agent's input content.

[0052] In one possible manner, the service scope description information corresponding to the agent is obtained as follows: based on the fifth model, the agent's configuration file is semantically parsed according to a preset service scope description information template to obtain the initial description information corresponding to the agent. The preset service scope description information template includes a sub-template corresponding to at least one of the description information used to generate the agent's functional positioning, skill range, and safety restriction rules; the initial description information is formatted according to a preset format to obtain the service scope description information corresponding to the agent in a preset format.

[0053] It should be noted that the configuration file of the intelligent agent may include content describing the functional definition, skill range and security restriction rules of the intelligent agent, which can be determined according to the actual scenario, and this disclosure does not impose any restrictions on it.

[0054] In this embodiment, the configuration file of the intelligent agent can be the system prompt words of the large model associated with the intelligent agent. The system prompt words can include content describing the functional definition of the large model, such as "You are a weather forecasting expert", and can also include content describing the skill range of the large model, such as "Skill 1: Predict tomorrow's weather based on meteorological data; Skill 2: Determine abnormal weather warnings based on meteorological data;...". It can also include content describing the security restriction rules of the large model, such as the security restriction conditions that the large model needs to follow during interaction, etc. The specific details can be determined according to the actual business scenario, and this disclosure does not impose any restrictions on this.

[0055] For example, the fifth model is used to semantically parse the input configuration file based on a preset service scope description information template, generate service scope description information for the agent, and output it according to a preset format. For example... Figure 2 As shown, configuration files and preset service scope description information templates can be input into the fifth model. The fifth model can deeply mine the agent's intent based on the preset service scope description information template, automatically reconstruct the semantics of the configuration information in the configuration file, eliminate ambiguity and strengthen the expression of key constraints, and generate standardized initial description information.

[0056] It is important to note that, generally speaking, the configuration file will at least include the function definition. If the configuration file does not include the skill scope and / or safety restriction rules, the large model can also expand the content of the skill scope and / or safety restriction rules according to the configuration information corresponding to the function definition. Specifically, the large model can be trained by building the corresponding training samples to obtain the ability to expand the content.

[0057] Furthermore, continue to refer to Figure 2Based on the initial description information, deep semantic analysis can be performed through the fifth model to extract the core functional definitions, skill ranges, and restriction rules of the agent. The content extracted from the initial description information is then formatted according to a preset format to obtain the service scope description information corresponding to the agent, such as JSON format service scope description information, or other formats that can be recognized by the large model. The specific format can be set according to the requirements, and this disclosure does not impose any restrictions on it.

[0058] By deeply understanding the agent's configuration file through a large model, the agent's intent can be accurately extracted, improving the accuracy and reliability of intent recognition. Furthermore, a standardized and unified format of service scope description information for the agent can be constructed, facilitating subsequent anomaly detection of the agent's input content based on the service scope description information, thereby improving the accuracy and reliability of anomaly detection.

[0059] Among possible approaches, the agent abnormal input detection method also includes: after the agent is updated, generating new service scope description information based on the configuration file corresponding to the updated agent; obtaining new input content for the agent; performing semantic understanding on the new input content and the new service scope description information based on the second major model; and determining the abnormal detection result corresponding to the new input content based on the semantic difference between the new input content and the new service scope description information.

[0060] For example, the agent or the large model associated with the agent can be iteratively updated as the actual business logic dynamically changes, and the service scope description information will also be updated accordingly. In other words, after the agent or the large model associated with the agent is updated, new service scope description information can be generated based on the new configuration file. Subsequently, anomaly detection of the agent's input content is performed based on this new service scope description information. This allows the detection mechanism based on service scope description information to overcome the limitations of static rule bases, achieve intelligent synchronization between detection strategies and business scenarios, and improve the accuracy and reliability of anomaly detection of the agent's input content.

[0061] In addition, semantic-level anomaly detection can significantly improve the detection capability of complex attacks such as variant expressions and sentence reconstruction, reduce the risk of detection mechanisms being bypassed, significantly enhance the adaptability and predictive ability to new attacks, and provide more reliable security for intelligent agents.

[0062] In possible approaches, the second major model includes a third major model and a fourth major model, with the fourth major model having more parameters than the third major model. Based on the second major model, semantic understanding is performed on the input content and the service range description information corresponding to the agent. Based on the semantic differences between the input content and the service range description information, anomaly detection results corresponding to the input content are determined. This includes: based on the third major model, semantic understanding is performed on the input content and the service range description information corresponding to the agent; based on the semantic differences between the input content and the service range description information, a first detection sub-result corresponding to the input content is determined. If the first detection sub-result indicates that the input content deviates from the service range description information, at least the input content and preset anomaly rules are semantically understood based on the fourth major model; based on the semantic differences between the input content and the preset anomaly rules, anomaly detection results corresponding to the input content are determined.

[0063] In this embodiment, the second major model used for anomaly detection may include a third major model and a fourth major model. The fourth major model has more parameters than the third major model, therefore the response speed of the third major model is faster than that of the fourth major model, but the judgment accuracy of the third major model is lower than that of the fourth major model. Based on this, the third major model can be used to quickly detect whether the input content is abnormal, while the fourth major model can be used for deep detection of whether the input content is abnormal.

[0064] For example, such as Figure 2 As shown, the input content and the service scope description information corresponding to the agent can first be input into the lightweight third model, so that the third model can quickly detect the input content based on the service scope description information and obtain the first detection sub-result corresponding to the input content.

[0065] Furthermore, if the first detection sub-result indicates that the input content deviates from the service range description information, a high-precision fourth model is used to perform deep detection on the input content based on at least preset anomaly rules to obtain the anomaly detection result corresponding to the input content. If the first detection sub-result indicates that the input content does not deviate from the service range description information, then the input content can be determined to be normal input content, and thus the input content can be directly filtered.

[0066] The hierarchical collaborative anomaly detection architecture based on lightweight large models and high-precision large models can achieve millisecond-level initial screening and condition-triggered deep verification through a fast detection process. While maintaining millisecond-level response speed, it ensures accurate identification of complex attacks, balances performance requirements, breaks through the technical bottleneck of the incompatibility between detection speed and detection accuracy, and achieves the unity of efficient resource utilization and comprehensive security protection.

[0067] It is worth noting that the fifth and fourth major models can be the same major model, and the corresponding model capabilities can be implemented according to different prompt words. Of course, the fifth and fourth major models can also be different major models, which can be set according to the needs. This disclosure does not impose any restrictions on this.

[0068] In possible ways, the anomaly detection result corresponding to the input content is determined based on the semantic difference between the input content and the preset anomaly rules, including: when there is a target anomaly rule in the preset anomaly rules that semantically matches the input content, the anomaly detection result representing that the input content matches the target anomaly rule is determined.

[0069] For example, such as Figure 3 As shown, after the preset anomaly rules are converted into a format recognizable by the large model, the fourth large model performs semantic matching between the input content and each preset anomaly rule. If a target anomaly rule exists among the preset anomaly rules that semantically matches the input content, the input content is determined to be an anomaly that conforms to the target anomaly rule. If no target anomaly rule semantically matches the input content, the input content can be determined to be normal input content, and thus the input content can be directly filtered.

[0070] It should be noted that if multiple semantically matching anomaly rules exist for the input content, the anomaly rule with the highest matching degree will be determined as the target anomaly rule. This allows the large model to perform context-aware semantic parsing of the input content based on preset anomaly rules, identifying complex threats such as variant attacks and combination attacks, achieving deep detection of the input content, and accurately identifying specific anomalies.

[0071] In one possible approach, based on the fourth major model, at least semantic understanding is performed on the input content and preset anomaly rules, and the anomaly detection result corresponding to the input content is determined based on the semantic difference between the input content and the preset anomaly rules. This includes: processing the input content, service scope description information, and preset anomaly rules based on the fourth major model as follows: performing semantic understanding on the input content and service scope description information, and determining the second detection sub-result corresponding to the input content based on the semantic difference between the input content and the service scope description information; when the second detection sub-result indicates that the input content deviates from the service scope description information, performing semantic understanding on the input content and preset anomaly rules, and determining the anomaly detection result indicating that the input content matches the target anomaly rule if there is a target anomaly rule in the preset anomaly rules that semantically matches the input content.

[0072] For example, such as Figure 2As shown, the high-precision fourth model can first correct the input content that deviates from the service range description information in the lightweight third model, thereby reducing the probability of misjudging the input content. Since this is a secondary judgment performed on the input content that deviates from the service range description information by the third model, normal input content has already been filtered out, greatly reducing the amount of data to be processed and having little impact on the model's processing performance.

[0073] For example, continue to refer to Figure 2 The fourth model with high precision performs secondary deviation detection on the input content based on the service range description information. If the input content deviates from the service range description information, the fourth model performs semantic matching between the input content and each preset anomaly rule. If there is a target anomaly rule in the preset anomaly rules that semantically matches the input content, the input content is determined to be an abnormal input content that conforms to the target anomaly rule.

[0074] Therefore, based on semantic-level deviation detection, the contextual semantics and potential attack intentions of the input content can be deeply analyzed, enabling effective identification of advanced adversarial techniques such as semantic confusion attacks and prompt word injection attacks, reducing the probability of misjudging the input content, and significantly improving the adversarial and generalization capabilities of the detection system.

[0075] In possible approaches, determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service scope description information includes: determining an anomaly detection result representing the input content deviating from the service scope description information when the semantic difference between the input content and the service scope description information satisfies at least one of the following conditions: the semantic matching degree between the input content and the first description information in the service scope description information is less than or equal to a first preset matching degree, where the first description information is used to describe the functional positioning of the agent; the semantic matching degree between the input content and the second description information in the service scope description information is less than or equal to a second preset matching degree, where the second description information is used to describe the skill range of the agent; the semantic matching degree between the input content and the third description information in the service scope description information is greater than a third preset matching degree, where the third description information is used to describe the security restriction rules of the agent.

[0076] For example, such as Figure 2 As shown, whether it is the rapid detection based on service scope description information of the lightweight large model or the secondary detection based on service scope description information of the high-precision large model, deviation detection of input content can be performed from multiple dimensions such as functional positioning, skill scope, and security restriction rules, thereby improving the coverage of anomaly detection, reducing the risk of security protection being bypassed, and improving the accuracy and reliability of anomaly detection.

[0077] The first, second, and third preset matching degrees can be set according to requirements, and this disclosure does not impose any restrictions on them. This allows for the detection and verification of whether the intent of the input content conforms to the functional definition of the agent based on functional positioning deviation, whether the intent of the input content conforms to the skill range of the agent based on skill range deviation, and whether the intent of the input content violates the agent's security restriction rules, such as sensitive instructions or unauthorized operations, based on security restriction rules.

[0078] It's worth noting that, to improve the efficiency of the aforementioned deviation detection, a binary deviation determination result can be output. For example, 1 indicates deviation, and 0 indicates no deviation. That is, if any deviation detection result is a deviation, the input content is determined to be abnormal input content that deviates from the service range description information. Furthermore, a detailed anomaly analysis report can be omitted to improve response speed.

[0079] In possible ways, the anomaly detection result includes at least one of the following: anomaly description of the input content, anomaly classification of the target anomaly rule, and anomaly level of the target anomaly rule, wherein the anomaly description of the input content is determined based on the input content and the rule description of the target anomaly rule.

[0080] For example, such as Figure 3 As shown, when the input content is determined to be abnormal, a detailed anomaly analysis report can be output based on the input content and the matching target anomaly rules. That is, the above-mentioned high-precision fourth model can directly output a detailed anomaly analysis report.

[0081] For example, the anomaly category of the input content can be determined based on the anomaly category corresponding to the target anomaly rule; the anomaly level of the input content can be determined based on the anomaly level corresponding to the target anomaly rule; and a detailed anomaly description can be determined based on the input content and the rule description of the target anomaly rule, etc. Here, the anomaly level characterizes the degree of anomaly of the input content; for example, a higher level indicates a greater security threat to the intelligent agent. This disclosure does not impose any limitations on this. This helps users understand the details of the anomalies in the input content and improves the readability and understandability of the anomaly detection results.

[0082] It should be noted that the anomaly analysis report may not include the analysis of normal input content in order to reduce data redundancy.

[0083] In some possible ways, the agent's abnormal input detection method also includes: refusing to input the input content into the agent's associated first model if the abnormal detection result corresponding to the input content meets at least one of the following conditions: the abnormal detection result indicates that the input content is abnormal; the abnormality level corresponding to the input content in the abnormal detection result meets a preset level.

[0084] For example, certain conditions can be set to prevent abnormal input from being fed into the large model associated with the agent, providing more reliable security for the agent. For instance, if the input matches a target anomaly rule and is judged as abnormal, its input to the large model associated with the agent can be prevented. Alternatively, if the anomaly level of the target anomaly rule matched by the input meets a preset level, the input can also be prevented from being fed into the large model associated with the agent, such as blocking input with a high anomaly level. This allows for flexible determination of the interception strategy for the agent's input based on requirements, providing more reliable security for the agent.

[0085] By combining fast and deep detection strategies with anomaly blocking strategies, the system balances the detection speed and accuracy of input content. It can both block abnormal requests in real time and ensure low latency, achieving a balance between efficient resource utilization and comprehensive security protection.

[0086] Based on the same concept, embodiments of this disclosure also provide an intelligent agent abnormal input detection device based on a large model, such as... Figure 4 As shown, the intelligent agent abnormal input detection device 400 may include:

[0087] The acquisition module 401 is used to acquire the input content of the agent; wherein, the agent is associated with a first large model, and the first large model is used to output the model processing result according to the input content of the agent;

[0088] The determination module 402 is used to perform semantic understanding on the input content and the service scope description information corresponding to the agent based on the second model, and determine the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service scope description information; wherein, the service scope description information corresponding to the agent is determined based on the agent's configuration file, and the configuration file is at least used to describe the definition of the service functions provided by the agent.

[0089] Optionally, the determining module 402 is used to:

[0090] An anomaly detection result representing a deviation of the input content from the service range description information is determined when the semantic difference between the input content and the service range description information satisfies at least one of the following conditions:

[0091] The semantic matching degree between the input content and the first description information in the service range description information is less than or equal to the first preset matching degree, and the first description information is used to describe the functional positioning of the intelligent agent.

[0092] The semantic matching degree between the input content and the second description information in the service scope description information is less than or equal to the second preset matching degree, and the second description information is used to describe the skill scope of the intelligent agent;

[0093] The semantic matching degree between the input content and the third description information in the service scope description information is greater than the third preset matching degree. The third description information is used to describe the security restriction rules of the intelligent agent.

[0094] Optionally, the second major model includes a third major model and a fourth major model, wherein the fourth major model has more parameters than the third major model. The determining module 402 is used to: perform semantic understanding on the input content and the service range description information corresponding to the agent based on the third major model; determine a first detection sub-result corresponding to the input content based on the semantic difference between the input content and the service range description information; and, if the first detection sub-result indicates that the input content deviates from the service range description information, perform semantic understanding on at least the input content and a preset anomaly rule based on the fourth major model; and determine an anomaly detection result corresponding to the input content based on the semantic difference between the input content and the preset anomaly rule.

[0095] Optionally, the determining module 402 is used to:

[0096] If a target anomaly rule exists in the preset anomaly rules that semantically matches the input content, an anomaly detection result representing that the input content matches the target anomaly rule is determined.

[0097] Optionally, the determining module 402 is used to:

[0098] Based on the fourth major model, the input content, the service scope description information, and the preset anomaly rules are processed as follows:

[0099] Semantic understanding is performed on the input content and the service range description information, and a second detection sub-result corresponding to the input content is determined based on the semantic difference between the input content and the service range description information;

[0100] When the second detection sub-result indicates that the input content deviates from the service range description information, semantic understanding is performed on the input content and the preset anomaly rules, and if there is a target anomaly rule in the preset anomaly rules that semantically matches the input content, an anomaly detection result indicating that the input content matches the target anomaly rule is determined.

[0101] Optionally, the anomaly detection result includes at least one of the anomaly description of the input content, the anomaly classification of the target anomaly rule, and the anomaly level of the target anomaly rule, wherein the anomaly description of the input content is determined based on the input content and the rule description of the target anomaly rule.

[0102] Optionally, the intelligent agent abnormal input detection device 400 may further include a rejection module, the rejection module being used for:

[0103] If the anomaly detection result corresponding to the input content meets at least one of the following conditions, the input content shall be rejected from being input into the first major model associated with the agent:

[0104] The anomaly detection result indicates that the input content is abnormal input content;

[0105] The anomaly level corresponding to the input content in the anomaly detection result meets the preset level.

[0106] Optionally, the service scope description information corresponding to the intelligent agent is obtained in the following way:

[0107] Based on the fifth model, the configuration file of the intelligent agent is semantically parsed according to the preset service scope description information template to obtain the initial description information corresponding to the intelligent agent. The preset service scope description information template includes a sub-template for generating at least one description information of the intelligent agent's functional positioning, skill range and security restriction rules.

[0108] The initial description information is processed according to a preset format to obtain the service scope description information corresponding to the agent in the preset format.

[0109] Optionally, the intelligent agent abnormal input detection device 400 may further include an update module, the update module being used for:

[0110] After the agent is updated, new service scope description information is generated based on the configuration file corresponding to the updated agent.

[0111] Obtain new input content for the agent.

[0112] Based on the second major model, semantic understanding is performed on the new input content and the new service scope description information. Based on the semantic differences between the new input content and the new service scope description information, the anomaly detection result corresponding to the new input content is determined.

[0113] Based on the same concept, embodiments of this disclosure also provide a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of any of the above-described methods for detecting abnormal inputs of intelligent agents based on large models.

[0114] Based on the same concept, this disclosure also provides an electronic device that may include:

[0115] A storage device on which computer programs are stored;

[0116] A processing device is used to execute a computer program stored in a storage device to implement the steps of any of the above-described methods for detecting abnormal inputs by an agent based on a large model.

[0117] Based on the same concept, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for detecting abnormal inputs of intelligent agents based on large models.

[0118] The following is for reference. Figure 5 This diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The terminal devices in these embodiments may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0119] like Figure 5 As shown, electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of electronic device 500. Processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0120] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0121] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0122] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0123] In some implementations, communication can be conducted using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can be interconnected with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0124] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0125] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire input content to an agent; wherein the agent is associated with a first major model, the first major model being used to output model processing results based on the input content of the agent; perform semantic understanding on the input content and the service scope description information corresponding to the agent based on a second major model, and determine anomaly detection results corresponding to the input content based on the semantic differences between the input content and the service scope description information; wherein the service scope description information corresponding to the agent is determined based on the agent's configuration file, the configuration file being used to at least describe the definition of the service functions provided by the agent.

[0126] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules are not, in some cases, intended to limit the functionality of the module itself.

[0129] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0130] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0132] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0133] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for detecting abnormal input to an intelligent agent based on a large model, characterized in that, The method includes: Obtain input content for the agent; wherein, the agent is associated with a first large model, and the first large model is used to output model processing results based on the input content of the agent; Based on the second major model, semantic understanding is performed on the input content and the service scope description information corresponding to the agent. Based on the semantic difference between the input content and the service scope description information, the anomaly detection result corresponding to the input content is determined. The service scope description information corresponding to the agent is determined based on the agent's configuration file, which at least describes the definition of the service functions provided by the agent. The step of performing semantic understanding on the input content and the service range description information corresponding to the agent based on the second major model, and determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service range description information, includes: performing semantic understanding on the input content and the service range description information corresponding to the agent; determining the first detection sub-result corresponding to the input content based on the semantic difference between the input content and the service range description information; and, if the first detection sub-result indicates that the input content deviates from the service range description information, performing semantic understanding on at least the input content and preset anomaly rules, and if there is a target anomaly rule in the preset anomaly rules that semantically matches the input content, determining the anomaly detection result indicating that the input content matches the target anomaly rule.

2. The method for detecting abnormal input to an intelligent agent based on a large model according to claim 1, characterized in that, The step of determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service scope description information includes: An anomaly detection result representing a deviation of the input content from the service range description information is determined when the semantic difference between the input content and the service range description information satisfies at least one of the following conditions: The semantic matching degree between the input content and the first description information in the service range description information is less than or equal to the first preset matching degree, and the first description information is used to describe the functional positioning of the intelligent agent. The semantic matching degree between the input content and the second description information in the service scope description information is less than or equal to the second preset matching degree, and the second description information is used to describe the skill scope of the intelligent agent; The semantic matching degree between the input content and the third description information in the service scope description information is greater than the third preset matching degree. The third description information is used to describe the security restriction rules of the intelligent agent.

3. The method for detecting abnormal input to an intelligent agent based on a large model according to claim 1, characterized in that, The second major model includes a third major model and a fourth major model, wherein the fourth major model has a greater number of parameters than the third major model; The step of performing semantic understanding on the input content and the service range description information corresponding to the agent based on the second major model, and determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service range description information, includes: performing semantic understanding on the input content and the service range description information corresponding to the agent based on the third major model, and determining the first detection sub-result corresponding to the input content based on the semantic difference between the input content and the service range description information; if the first detection sub-result indicates that the input content deviates from the service range description information, performing semantic understanding on at least the input content and the preset anomaly rule based on the fourth major model, and determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the preset anomaly rule.

4. The method for detecting abnormal input to an intelligent agent based on a large model according to claim 3, characterized in that, The step of performing semantic understanding on the input content and preset anomaly rules based on the fourth model, and determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the preset anomaly rules, includes: Based on the fourth major model, the input content, the service scope description information, and the preset anomaly rules are processed as follows: Semantic understanding is performed on the input content and the service range description information, and a second detection sub-result corresponding to the input content is determined based on the semantic difference between the input content and the service range description information; When the second detection sub-result indicates that the input content deviates from the service range description information, semantic understanding is performed on the input content and the preset anomaly rules, and if there is a target anomaly rule in the preset anomaly rules that semantically matches the input content, an anomaly detection result indicating that the input content matches the target anomaly rule is determined.

5. The method for detecting abnormal input to an intelligent agent based on a large model according to any one of claims 1-4, characterized in that, The anomaly detection result includes at least one of the anomaly description of the input content, the anomaly classification of the target anomaly rule, and the anomaly level of the target anomaly rule, wherein the anomaly description of the input content is determined based on the input content and the rule description of the target anomaly rule.

6. The method for detecting abnormal input to an intelligent agent based on a large model according to any one of claims 1-4, characterized in that, The method further includes: If the anomaly detection result corresponding to the input content meets at least one of the following conditions, the input content will be rejected from being input into the first major model associated with the agent: The anomaly detection result indicates that the input content is abnormal input content; The anomaly level corresponding to the input content in the anomaly detection result meets the preset level.

7. The method for detecting abnormal input to an intelligent agent based on a large model according to any one of claims 1-4, characterized in that, The service scope description information corresponding to the intelligent agent is obtained in the following way: Based on the fifth model, the configuration file of the intelligent agent is semantically parsed according to the preset service scope description information template to obtain the initial description information corresponding to the intelligent agent. The preset service scope description information template includes a sub-template for generating at least one description information of the intelligent agent's functional positioning, skill range and security restriction rules. The initial description information is processed according to a preset format to obtain the service scope description information corresponding to the agent in the preset format.

8. The method for detecting abnormal input to an intelligent agent based on a large model according to any one of claims 1-4, characterized in that, The method further includes: After the agent is updated, new service scope description information is generated based on the configuration file corresponding to the updated agent. Obtain new input content for the agent. Based on the second major model, semantic understanding is performed on the new input content and the new service scope description information. Based on the semantic differences between the new input content and the new service scope description information, the anomaly detection result corresponding to the new input content is determined.

9. A device for detecting abnormal input to an intelligent agent based on a large model, characterized in that, The device includes: An acquisition module is used to acquire input content to the agent; wherein, the agent is associated with a first large model, and the first large model is used to output model processing results based on the input content of the agent; The determination module is used to perform semantic understanding on the input content and the service scope description information corresponding to the agent based on the second major model, and to determine the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service scope description information; wherein, the service scope description information corresponding to the agent is determined based on the agent's configuration file, and the configuration file is at least used to describe the definition of the service functions provided by the agent; The determining module is configured to: perform semantic understanding on the input content and the service range description information corresponding to the agent; determine a first detection sub-result corresponding to the input content based on the semantic difference between the input content and the service range description information; and, if the first detection sub-result indicates that the input content deviates from the service range description information, perform semantic understanding on at least the input content and preset anomaly rules; and, if there is a target anomaly rule in the preset anomaly rules that semantically matches the input content, determine an anomaly detection result indicating that the input content matches the target anomaly rule.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processing device, the computer program performs the steps of the method according to any one of claims 1-8.

11. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images