Intelligent agent abnormal input detection method and device based on large model and electronic equipment

Abstract

The invention discloses an agent abnormal input detection method and device based on a large model and electronic equipment, and relates to the technical field of large models, agents and artificial intelligence. The method comprises the following steps: acquiring input content of an intelligent agent; wherein the intelligent agent is associated with a first large model, and the first large model is used for outputting a model processing result according to the input content of the intelligent agent; performing semantic understanding on the input content and service range description information corresponding to the intelligent agent based on a second large model, and determining an anomaly detection result corresponding to the input content according to a semantic difference between the input content and the service range description information; wherein the service range description information corresponding to the intelligent agent is determined based on a configuration file of the intelligent agent, and the configuration file is at least used for describing the definition of a service function provided by the intelligent agent. Therefore, abnormal input of semantic deformation means such as synonym replacement, sentence pattern reconstruction and interference word addition can be effectively recognized, and the accuracy and reliability of abnormal detection on the input content of the intelligent agent are improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of large models, intelligent agents, and artificial intelligence technology, and in particular, to a method, device, and electronic device for detecting abnormal input of an intelligent agent based on a large model. Background Art

[0002] Large-scale intelligent agents are widely used in areas such as intelligent customer service, automated decision-making, and intelligent assistants. Their core capability is to understand and execute complex tasks through natural language interaction. With the diversification of interaction scenarios, intelligent agents face increasingly hidden and semantic security threats.

[0003] In related technologies, detection methods based on keyword matching or static template matching cannot identify abnormal inputs that use semantic deformation methods such as synonym replacement, sentence reconstruction, and the addition of interference words. There is a possibility that the detection mechanism will fail, resulting in low accuracy and reliability of anomaly detection. Summary of the Invention

[0004] This summary is provided to briefly introduce concepts that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a method for detecting abnormal input of an intelligent agent based on a large model, the method comprising: Obtaining input content for an agent; wherein the agent is associated with a first large model, and the first large model is used to output a model processing result based on the input content of the agent; Based on the second largest model, semantic understanding is performed on the input content and the service scope description information corresponding to the intelligent agent, and according to the semantic difference between the input content and the service scope description information, the anomaly detection result corresponding to the input content is determined; wherein, the service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is at least used to describe the definition of the service function provided by the intelligent agent.

[0006] In a second aspect, the present disclosure provides a large-model-based intelligent agent abnormal input detection device, the device comprising: An acquisition module, configured to acquire input content for an agent; wherein the agent is associated with a first large model, and the first large model is configured to output a model processing result based on the input content of the agent; A determination module is used to perform semantic understanding of the input content and the service scope description information corresponding to the intelligent agent based on the second largest model, and determine the anomaly detection result corresponding to the input content according to the semantic difference between the input content and the service scope description information; wherein, the service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is at least used to describe the definition of the service function provided by the intelligent agent.

[0007] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when executed by a processing device.

[0008] In a fourth aspect, the present disclosure provides an electronic device, comprising: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method in the first aspect.

[0009] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0010] Through the above technical solution, a large-scale model is used to semantically understand the agent's input content and the agent's corresponding service scope description information. Based on the semantic differences between the input content and the service scope description information, anomaly detection results corresponding to the input content are determined. This method, which uses a large-scale model to semantically understand the agent's service scope description information and input content, can detect anomalies in the input content based on at least the semantic differences between the input intent and the service functions provided by the agent. This effectively identifies anomalous inputs that use semantic distortion methods such as synonym replacement, sentence reconstruction, and the addition of noise words, thereby improving the accuracy and reliability of anomaly detection for the agent's input content.

[0011] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1 is a schematic flow chart of a method for detecting abnormal input of an intelligent agent based on a large model according to an exemplary embodiment of the present disclosure; Figure 2 This is a process diagram of a method for detecting abnormal input of an intelligent agent according to an exemplary embodiment of the present disclosure; Figure 3 is a schematic diagram showing a preset exception rule according to an exemplary embodiment of the present disclosure; Figure 4 1 is a schematic structural diagram of a device for detecting abnormal input of an intelligent agent based on a large model according to an exemplary embodiment of the present disclosure; Figure 5 The figure is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0013] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0014] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0015] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0017] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0018] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0019] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0020] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0021] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0023] At the same time, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] With the diversified development of intelligent agent interaction scenarios, the security threats they face are becoming more hidden and semantic. For example, constructed abnormal inputs can be used to guide intelligent agents to perform unauthorized operations, leak sensitive data, or generate harmful content, which poses a serious threat to user privacy protection and corporate asset security.

[0025] In related technologies, anomaly detection methods mainly use the following input detection mechanisms: (1) Rule-based keyword matching: String matching is performed using a predefined keyword library (such as malicious instructions and privacy fields), for example, using regular expressions for matching operations. This method is effective for explicitly abnormal statements, but cannot identify semantically equivalent variant expressions such as synonym replacement and grammatical reconstruction.

[0026] (2) Static template matching: A static template library of normal input content is pre-established, and the input content is judged to be abnormal by comparing the syntactic structure. This method is suitable for input scenarios with fixed sentence patterns, but it will produce a large number of false positives when processing open-domain dialogues, and it is difficult to identify semantic confusion attacks implemented by adding interference words, misleading context, etc.

[0027] (3) Single-dimensional intent classification: Use the natural language processing model to classify the input content and determine whether the input content is abnormal by comparing it with the preset categories. Model processing takes a certain amount of time and has a slow response speed. In addition, if the input content does not fall within the preset category, or if the intent categories overlap or are nested, the detection will fail.

[0028] Among the aforementioned related technologies, detection methods based on keyword matching and static template matching are unable to effectively identify semantic obfuscation attacks, resulting in weak detection capabilities, making detection mechanisms easily bypassed and security protection ineffective. Detection methods based on keyword matching, static template matching, and fixed intent classification struggle to adapt to dynamic changes in agent business logic, such as feature updates and policy adjustments, as well as the continuous evolution of attack methods, leading to insufficient detection coverage and high false positive rates.

[0029] In view of this, the present disclosure provides a method, device and electronic device for detecting abnormal input of an intelligent agent based on a large model to solve the above technical problems.

[0030] The following further explains the embodiments of the present disclosure with reference to the accompanying drawings.

[0031] Figure 1 This is a flowchart of a method for detecting abnormal input of an intelligent agent based on a large model according to an exemplary embodiment of the present disclosure, with reference to Figure 1 , the agent abnormal input detection method may include the following steps: S101: Obtain input content for the agent.

[0032] Among them, the intelligent agent is associated with a first large model, and the first large model is used to output the model processing result according to the input content of the intelligent agent.

[0033] For example, the intelligent agent can be an intelligent agent that accesses the platform to realize intelligent customer service, automated decision-making, intelligent assistant and other capabilities. Users can interact with the intelligent agent through the intelligent interactive page provided by the platform. Taking the intelligent customer service scenario as an example, the large model associated with the intelligent agent can output the conversation content based on the user's input content, and the input content of the intelligent agent is the input content of the user on the conversation page. It can be set according to needs, and this disclosure does not limit this.

[0034] S102: Perform semantic understanding on the input content and the service scope description information corresponding to the agent based on the second largest model, and determine the anomaly detection result corresponding to the input content according to the semantic difference between the input content and the service scope description information.

[0035] The service scope description information corresponding to the agent is determined based on a configuration file of the agent, and the configuration file is at least used to describe the definition of the service function provided by the agent.

[0036] It should be noted that the first large model is the large model associated with the agent, which is determined based on the actual business scenario, while the second large model is the large model used to detect anomalies in the agent's input content. In other words, the first and second large models are different.

[0037] For example, the service scope description information corresponding to the agent is determined based on a configuration file that includes at least a definition for describing the service functions provided by the agent. Therefore, by performing semantic understanding on the service scope description information corresponding to the agent through the big model, at least the service functions actually provided by the agent can be obtained. By performing semantic understanding on the input content through the big model, it can be determined that the service functions that the input content expects the agent to provide can be determined. By comparing whether the actual service functions match the expected service functions, it can be determined whether the input content is abnormal input content. For example, the big model associated with the agent is a big model for querying the weather, and the input content is "Query order xx". The service functions actually provided by the big model do not match the service functions that the input content expects the agent to provide, then the input content is abnormal input content.

[0038] By adopting the above method, the service scope description information and input content of the intelligent agent are semantically understood based on the large model, and at least anomaly detection of the input content can be performed based on the semantic differences between the input intention of the input content and the service functions provided by the intelligent agent, thereby effectively identifying abnormal inputs with semantic deformation methods such as synonym replacement, sentence reconstruction, and adding interference words, thereby improving the accuracy and reliability of anomaly detection of the intelligent agent's input content.

[0039] In a possible manner, the service scope description information corresponding to the intelligent agent is obtained in the following manner: based on the fifth model, the configuration file of the intelligent agent is semantically parsed according to a preset service scope description information template to obtain the initial description information corresponding to the intelligent agent, and the preset service scope description information template includes a sub-template corresponding to at least one description information for generating the functional positioning, skill range and security restriction rules of the intelligent agent; the initial description information is formatted according to a preset format to obtain the service scope description information corresponding to the intelligent agent in the preset format.

[0040] It should be noted that the configuration file of the intelligent agent may include content such as the function definition, skill range and safety restriction rules for describing the intelligent agent, which can be determined according to the actual scenario and is not limited by this disclosure.

[0041] In this embodiment, the configuration file of the intelligent agent can be the system prompt word of the large model associated with the intelligent agent. The system prompt word may include content used to describe the functional definition of the large model, such as "You are an expert in weather forecasting", and may also include content used to describe the skill range of the large model, such as "Skill 1: Predict tomorrow's weather conditions based on meteorological data; Skill 2: Determine abnormal weather warnings based on meteorological data;...", and may also include content used to describe the safety restriction rules of the large model, such as the safety restriction conditions that the large model needs to follow during the interaction process, etc. The specific content can be determined based on the actual business scenario, and this disclosure does not impose any restrictions on this.

[0042] For example, the fifth model is used to perform semantic parsing on the input configuration file according to the preset service scope description information template, generate the service scope description information of the agent and output it in the preset format. Figure 2 As shown, the configuration file and the preset service scope description information template can be input into the fifth model. The fifth model can deeply mine the intention of the intelligent agent based on the preset service scope description information template, automatically reconstruct the semantics of the configuration information in the configuration file, eliminate ambiguity and strengthen the key constraint expression, and generate standardized initial description information.

[0043] It should be noted that, generally speaking, the configuration file will at least include the function definition. If the configuration file does not include the skill range and / or safety restriction rules, the large model can also expand the content of the skill range and / or safety restriction rules based on the configuration information corresponding to the function definition. Specifically, the large model can be trained based on the construction of corresponding training samples to obtain the ability to expand content.

[0044] Further, continue to refer to Figure 2 Based on the initial description information, the fifth large model can continue to perform in-depth semantic analysis to extract the core functional definition, skill range, restriction rules and other contents of the intelligent agent, and format the content extracted from the initial description information according to the preset format to obtain the service scope description information corresponding to the intelligent agent, such as the service scope description information in JSON format, or other formats that can be recognized by the large model. The specific settings can be based on the needs, and this disclosure does not impose any restrictions on this.

[0045] Through the large model, we can deeply understand the configuration file of the intelligent agent, accurately extract the intention of the intelligent agent, improve the accuracy and reliability of the recognition of the intelligent agent's intention, and construct the service scope description information of the intelligent agent in a standard and unified format, so as to facilitate the subsequent anomaly detection of the intelligent agent's input content based on the service scope description information, thereby improving the accuracy and reliability of anomaly detection of the intelligent agent's input content.

[0046] In a possible manner, the method for detecting abnormal input of an intelligent agent also includes: after the intelligent agent is updated, generating new service scope description information based on the configuration file corresponding to the updated intelligent agent; obtaining new input content for the intelligent agent, performing semantic understanding of the new input content and the new service scope description information based on the second largest model, and determining the abnormal detection result corresponding to the new input content based on the semantic difference between the new input content and the new service scope description information.

[0047] For example, an agent or its associated macromodel can be iteratively updated as actual business logic changes, and accordingly, the service scope description information will be updated accordingly. In other words, after an agent or its associated macromodel is updated, a new service scope description information can be generated based on the new configuration file. Subsequently, anomaly detection of the agent's input content can be performed based on this new service scope description information. This service scope description-based detection mechanism can thus transcend the limitations of a static rule base, achieve intelligent synchronization of detection strategies with business scenarios, and improve the accuracy and reliability of anomaly detection of the agent's input content.

[0048] In addition, anomaly detection based on the semantic level can significantly improve the detection capabilities of complex attacks such as variant expressions and sentence reconstruction, reduce the risk of detection mechanisms being bypassed, significantly enhance the adaptability and prediction capabilities to new attacks, and provide more reliable security protection for intelligent entities.

[0049] In a possible manner, the second largest model includes a third largest model and a fourth largest model, and the number of parameters of the fourth largest model is greater than the number of parameters of the third largest model. Performing semantic understanding of the input content and the service scope description information corresponding to the intelligent agent based on the second largest model, and determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the service scope description information, including: performing semantic understanding of the input content and the service scope description information corresponding to the intelligent agent based on the third largest model, and determining the first detection sub-result corresponding to the input content based on the semantic difference between the input content and the service scope description information; when the first detection sub-result indicates that the input content deviates from the service scope description information, performing semantic understanding of at least the input content and a preset anomaly rule based on the fourth largest model, and determining the anomaly detection result corresponding to the input content based on the semantic difference between the input content and the preset anomaly rule.

[0050] In this embodiment, the second largest model used for anomaly detection can include the third largest model and the fourth largest model. The fourth largest model has more parameters than the third largest model, so the third largest model responds faster than the fourth largest model, but the third largest model has lower judgment accuracy than the fourth largest model. Based on this, the third largest model can be used to quickly detect whether the input content is abnormal, and the fourth largest model can be used to deeply detect whether the input content is abnormal.

[0051] For example, Figure 2 As shown, the input content and the service scope description information corresponding to the intelligent agent can first be input into the lightweight third model, so that the third model can quickly detect the input content based on the service scope description information to obtain the first detection sub-result corresponding to the input content.

[0052] Furthermore, if the first detection sub-result indicates that the input content deviates from the service scope description information, a high-precision fourth model is used to perform in-depth detection of the input content based on at least preset anomaly rules to obtain an anomaly detection result corresponding to the input content. If the first detection sub-result indicates that the input content does not deviate from the service scope description information, the input content can be determined to be normal input content and can be directly filtered.

[0053] The hierarchical and collaborative anomaly detection architecture based on lightweight large models and high-precision large models can achieve millisecond-level initial screening and condition-triggered deep verification through a fast detection process. While maintaining a millisecond-level response speed, it ensures the accurate identification of complex attacks, balances performance requirements, breaks through the technical bottleneck of the inability to achieve both detection speed and detection accuracy, and realizes the unity of efficient resource utilization and comprehensive security protection.

[0054] It is worth noting that the fifth model and the fourth model can be the same model, and the corresponding model capabilities can be realized according to different prompt words. Of course, the fifth model and the fourth model can also be different models, which can be set according to specific needs. This disclosure does not impose any restrictions on this.

[0055] In a possible manner, determining an anomaly detection result corresponding to the input content based on the semantic difference between the input content and the preset anomaly rule includes: when there is a target anomaly rule in the preset anomaly rule that semantically matches the input content, determining an anomaly detection result that characterizes that the input content matches the target anomaly rule.

[0056] For example, Figure 3 As shown, after the preset exception rules are converted into a format recognizable by the large model, the input content is semantically matched with each preset exception rule through the fourth large model. If there is a target exception rule among the preset exception rules that semantically matches the input content, the input content is determined to be abnormal input content that meets the target exception rule. If there is no target exception rule that semantically matches the input content, the input content can be determined to be normal input content and can be directly filtered.

[0057] It should be noted that if the input content has multiple semantically matching anomaly rules, the anomaly rule with the highest match is determined as the target anomaly rule. This allows the large model to perform context-aware semantic parsing of the input content based on preset anomaly rules, identifying complex threats such as variant attacks and combined attacks, achieving in-depth detection of the input content, and accurately identifying specific anomalies in the input content.

[0058] In a possible manner, based on the fourth model, at least the input content and the preset exception rules are semantically understood, and according to the semantic difference between the input content and the preset exception rules, the exception detection result corresponding to the input content is determined, including: based on the fourth model, the input content, the service scope description information and the preset exception rules are processed as follows: the input content and the service scope description information are semantically understood, and according to the semantic difference between the input content and the service scope description information, the second detection sub-result corresponding to the input content is determined; when the second detection sub-result represents that the input content deviates from the service scope description information, the input content and the preset exception rules are semantically understood, and when there is a target exception rule in the preset exception rules that semantically matches the input content, the exception detection result representing that the input content matches the target exception rule is determined.

[0059] For example, Figure 2 As shown, the high-precision fourth model can first be used to correct input content that deviates from the service scope description, thereby reducing the probability of misjudgment of input content. Because this secondary judgment is performed on input content that deviates from the service scope description, the correct input content has been filtered out, significantly reducing the amount of data to be processed and minimizing the impact on model processing performance.

[0060] For example, continue to refer to Figure 2 , a secondary deviation detection is performed on the input content based on the service scope description information through the high-precision fourth model. When it is determined that the input content deviates from the service scope description information, the input content is semantically matched with each preset exception rule through the fourth model. When there is a target exception rule in the preset exception rules that semantically matches the input content, the input content is determined to be abnormal input content that complies with the target exception rule.

[0061] Therefore, based on semantic-level deviation detection, we can deeply analyze the contextual semantics and potential attack intentions of the input content, effectively identify advanced adversarial methods such as semantic confusion attacks and prompt word injection attacks, reduce the probability of misjudgment of input content, and significantly improve the adversarial and generalization capabilities of the detection system.

[0062] In a possible manner, based on the semantic difference between the input content and the service scope description information, determining the anomaly detection result corresponding to the input content, including: determining the anomaly detection result characterizing that the input content deviates from the service scope description information when the semantic difference between the input content and the service scope description information satisfies at least one of the following conditions: the semantic matching degree between the input content and the first description information in the service scope description information is less than or equal to the first preset matching degree, and the first description information is used to describe the functional positioning of the intelligent body; the semantic matching degree between the input content and the second description information in the service scope description information is less than or equal to the second preset matching degree, and the second description information is used to describe the skill range of the intelligent body; the semantic matching degree between the input content and the third description information in the service scope description information is greater than the third preset matching degree, and the third description information is used to describe the safety restriction rules of the intelligent body.

[0063] For example, Figure 2 As shown, whether it is the rapid detection based on the service scope description information by the lightweight large model or the secondary detection based on the service scope description information by the high-precision large model, deviation detection of input content can be performed from multiple dimensions such as functional positioning, skill scope, and security restriction rules, thereby improving the coverage of anomaly detection, reducing the risk of security protection being bypassed, and improving the accuracy and reliability of anomaly detection.

[0064] The first preset matching degree, the second preset matching degree, and the third preset matching degree can be set as needed, and this disclosure does not impose any restrictions on this. In this way, the intention of the input content can be verified based on the function positioning deviation detection to see whether it conforms to the function definition of the intelligent agent, the intention of the input content can be verified based on the skill range deviation detection to see whether it conforms to the skill range of the intelligent agent, and the intention of the input content can be verified based on the security restriction rule deviation detection to see whether it violates the security restriction rules of the intelligent agent, such as sensitive instructions, unauthorized operations, etc.

[0065] It's worth noting that to improve the efficiency of the aforementioned deviation detection, a binary deviation determination result can be output, for example, 1 indicating deviation and 0 indicating no deviation. That is, if any deviation detection result indicates a deviation, the input content is considered abnormal, deviating from the service scope description information. Furthermore, detailed anomaly analysis reports can be omitted to improve response speed.

[0066] In a possible manner, the anomaly detection result includes at least one of an anomaly description of the input content, an anomaly classification of the target anomaly rule, and an anomaly level of the target anomaly rule, wherein the anomaly description of the input content is determined based on the input content and the rule description of the target anomaly rule.

[0067] For example, Figure 3As shown, when the input content is determined to be abnormal input content, a detailed abnormality analysis report can be output based on the input content and the matching target abnormality rules, that is, the above-mentioned high-precision fourth model can directly output a detailed abnormality analysis report.

[0068] For example, the abnormality classification of the input content can be determined based on the abnormality classification corresponding to the target abnormality rule, the abnormality level of the input content can be determined based on the abnormality level corresponding to the target abnormality rule, and a detailed abnormality description can be determined based on the input content and the rule description of the target abnormality rule, etc., wherein the abnormality level is used to represent the degree of abnormality of the input content. For example, the higher the level, the greater the security threat to the intelligent agent. This is not limited in this disclosure. This can help users understand the abnormal details of the input content and improve the readability and comprehensibility of the abnormality detection results.

[0069] It should be noted that the abnormal analysis report may not include analysis of normal input content to reduce data redundancy.

[0070] In a possible manner, the intelligent agent abnormal input detection method also includes: refusing to input the input content into the first large model associated with the intelligent agent when the abnormal detection result corresponding to the input content meets at least one of the following conditions: the abnormal detection result characterizes the input content as abnormal input content; the abnormal level corresponding to the input content in the abnormal detection result meets the preset level.

[0071] For example, certain conditions can be set to prevent abnormal input content from being input into the large model associated with the agent, providing more reliable security for the agent. For example, if the input content matches the target abnormality rule and is judged to be abnormal input content, the input content can be prevented from being input into the large model associated with the agent. Or, if the abnormality level corresponding to the target abnormality rule matched by the input content meets the preset level, the input content can be prevented from being input into the large model associated with the agent, for example, intercepting input content with a high abnormality level. This allows the interception strategy for the agent's input content to be flexibly determined according to demand, providing more reliable security for the agent.

[0072] Through the anomaly detection strategy and anomaly interception strategy that combine fast inspection and deep inspection, we take into account both the detection speed and accuracy of the input content, and can not only achieve real-time blocking of abnormal requests, but also ensure low latency requirements, and achieve the unity of efficient resource utilization and comprehensive security protection.

[0073] Based on the same concept, the embodiment of the present disclosure also provides an intelligent agent abnormal input detection device based on a large model, such as Figure 4 As shown, the intelligent agent abnormal input detection device 400 may include: An acquisition module 401 is configured to acquire input content for an agent, wherein the agent is associated with a first large model, and the first large model is configured to output a model processing result based on the input content of the agent; Determination module 402 is used to perform semantic understanding of the input content and the service scope description information corresponding to the intelligent agent based on the second largest model, and determine the anomaly detection result corresponding to the input content according to the semantic difference between the input content and the service scope description information; wherein, the service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is at least used to describe the definition of the service function provided by the intelligent agent.

[0074] Optionally, the determining module 402 is configured to: If the semantic difference between the input content and the service scope description information satisfies at least one of the following conditions, determining an anomaly detection result indicating that the input content deviates from the service scope description information: The semantic matching degree between the input content and the first description information in the service scope description information is less than or equal to a first preset matching degree, the first description information being used to describe the functional positioning of the agent; The semantic matching degree between the input content and the second description information in the service scope description information is less than or equal to a second preset matching degree, the second description information being used to describe the skill scope of the agent; The semantic matching degree between the input content and the third description information in the service scope description information is greater than a third preset matching degree, and the third description information is used to describe the security restriction rules of the agent.

[0075] Optionally, the second largest model includes a third largest model and a fourth largest model, and the parameter amount of the fourth largest model is greater than the parameter amount of the third largest model. The determination module 402 is used to: perform semantic understanding of the input content and the service scope description information corresponding to the intelligent agent based on the third largest model, and determine the first detection sub-result corresponding to the input content according to the semantic difference between the input content and the service scope description information; when the first detection sub-result represents that the input content deviates from the service scope description information, perform semantic understanding of at least the input content and the preset exception rule based on the fourth largest model, and determine the exception detection result corresponding to the input content according to the semantic difference between the input content and the preset exception rule.

[0076] Optionally, the determining module 402 is configured to: In a case where a target exception rule semantically matching the input content exists in the preset exception rules, an exception detection result indicating that the input content matches the target exception rule is determined.

[0077] Optionally, the determining module 402 is configured to: The input content, the service scope description information, and the preset exception rules are processed as follows based on the fourth model: Performing semantic understanding on the input content and the service scope description information, and determining a second detection sub-result corresponding to the input content based on a semantic difference between the input content and the service scope description information; When the second detection sub-result represents that the input content deviates from the service scope description information, semantic understanding is performed on the input content and the preset exception rules, and when there is a target exception rule in the preset exception rules that semantically matches the input content, an exception detection result representing that the input content matches the target exception rule is determined.

[0078] Optionally, the anomaly detection result includes at least one of an anomaly description of the input content, an anomaly classification of the target anomaly rule, and an anomaly level of the target anomaly rule, wherein the anomaly description of the input content is determined based on the input content and the rule description of the target anomaly rule.

[0079] Optionally, the agent abnormal input detection device 400 may further include a rejection module, wherein the rejection module is configured to: If the abnormality detection result corresponding to the input content meets at least one of the following conditions, the input content is refused to be input into the first model associated with the agent: The abnormality detection result indicates that the input content is abnormal input content; The abnormality level corresponding to the input content in the abnormality detection result meets the preset level.

[0080] Optionally, the service scope description information corresponding to the agent is obtained in the following manner: Based on the fifth model, semantically parsing the agent's configuration file according to a preset service scope description information template is performed to obtain initial description information corresponding to the agent, wherein the preset service scope description information template includes a sub-template for generating description information corresponding to at least one of the agent's functional positioning, skill scope, and security restriction rules; The initial description information is formatted according to a preset format to obtain the service scope description information corresponding to the agent in the preset format.

[0081] Optionally, the intelligent agent abnormal input detection device 400 may further include an updating module, wherein the updating module is configured to: After the agent is updated, new service scope description information is generated according to the configuration file corresponding to the updated agent; Obtaining new input to the agent, The new input content and the new service scope description information are semantically understood based on the second largest model, and the anomaly detection result corresponding to the new input content is determined according to the semantic difference between the new input content and the new service scope description information.

[0082] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processing device, the steps of any of the above-mentioned large-model-based intelligent agent abnormal input detection methods are implemented.

[0083] Based on the same concept, an embodiment of the present disclosure further provides an electronic device, which may include: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement any step of the above-mentioned large model-based intelligent agent abnormal input detection method.

[0084] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned large-model-based intelligent agent abnormal input detection methods.

[0085] Reference below Figure 5 , which shows a schematic structural diagram of an electronic device 500 suitable for implementing the embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0086] like Figure 5As shown, electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 502 or programs loaded from a storage device 508 into a random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of electronic device 500. Processing device 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0087] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0088] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0089] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0090] In some embodiments, communications may be conducted using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0091] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0092] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains input content for the intelligent agent; wherein the intelligent agent is associated with a first large model, and the first large model is used to output the model processing result according to the input content of the intelligent agent; based on the second large model, the input content and the service scope description information corresponding to the intelligent agent are semantically understood, and according to the semantic difference between the input content and the service scope description information, the abnormality detection result corresponding to the input content is determined; wherein the service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is at least used to describe the definition of the service function provided by the intelligent agent.

[0093] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules described in the embodiments of the present disclosure may be implemented in software or hardware, wherein the name of a module does not necessarily limit the module itself.

[0096] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0097] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0099] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0100] Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims. Regarding the apparatus in the above-described embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method and will not be elaborated upon here.

Claims

1. A method for detecting abnormal input of intelligent agents based on a large model, characterized in that: The method includes: Obtaining input content for an agent; wherein the agent is associated with a first large model, and the first large model is used to output a model processing result based on the input content of the agent; Based on the second largest model, semantic understanding is performed on the input content and the service scope description information corresponding to the intelligent agent, and according to the semantic difference between the input content and the service scope description information, the anomaly detection result corresponding to the input content is determined; wherein, the service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is at least used to describe the definition of the service function provided by the intelligent agent.

2. The method for detecting abnormal input of an intelligent agent based on a large model according to claim 1, characterized in that: The determining, based on the semantic difference between the input content and the service scope description information, an anomaly detection result corresponding to the input content includes: If the semantic difference between the input content and the service scope description information satisfies at least one of the following conditions, determining an anomaly detection result indicating that the input content deviates from the service scope description information: The semantic matching degree between the input content and the first description information in the service scope description information is less than or equal to a first preset matching degree, the first description information being used to describe the functional positioning of the agent; The semantic matching degree between the input content and the second description information in the service scope description information is less than or equal to a second preset matching degree, the second description information being used to describe the skill scope of the agent; The semantic matching degree between the input content and the third description information in the service scope description information is greater than a third preset matching degree, and the third description information is used to describe the security restriction rules of the agent.

3. The method for detecting abnormal input of an intelligent agent based on a large model according to claim 1, characterized in that: The second largest model includes a third largest model and a fourth largest model, and the number of parameters of the fourth largest model is greater than the number of parameters of the third largest model; The method of performing semantic understanding on the input content and the service scope description information corresponding to the agent based on the second largest model, and determining the abnormality detection result corresponding to the input content according to the semantic difference between the input content and the service scope description information, includes: performing semantic understanding on the input content and the service scope description information corresponding to the agent based on the third largest model, and determining the first detection sub-result corresponding to the input content according to the semantic difference between the input content and the service scope description information; when the first detection sub-result represents that the input content deviates from the service scope description information, performing semantic understanding on at least the input content and the preset abnormality rule based on the fourth largest model, and determining the abnormality detection result corresponding to the input content according to the semantic difference between the input content and the preset abnormality rule.

4. The method for detecting abnormal input of an intelligent agent based on a large model according to claim 3, characterized in that: The determining, based on the semantic difference between the input content and the preset exception rule, an anomaly detection result corresponding to the input content includes: In a case where a target exception rule semantically matching the input content exists in the preset exception rules, an exception detection result indicating that the input content matches the target exception rule is determined.

5. The method for detecting abnormal input of an intelligent agent based on a large model according to claim 3, characterized in that: The performing semantic understanding of at least the input content and the preset exception rule based on the fourth model and determining the exception detection result corresponding to the input content according to the semantic difference between the input content and the preset exception rule includes: The input content, the service scope description information, and the preset exception rules are processed as follows based on the fourth model: Performing semantic understanding on the input content and the service scope description information, and determining a second detection sub-result corresponding to the input content based on a semantic difference between the input content and the service scope description information; When the second detection sub-result represents that the input content deviates from the service scope description information, semantic understanding is performed on the input content and the preset exception rules, and when there is a target exception rule in the preset exception rules that semantically matches the input content, an exception detection result representing that the input content matches the target exception rule is determined.

6. The method for detecting abnormal input of an intelligent agent based on a large model according to claim 4 or 5, characterized in that: The anomaly detection result includes at least one of an anomaly description of the input content, an anomaly classification of the target anomaly rule, and an anomaly level of the target anomaly rule, wherein the anomaly description of the input content is determined based on the input content and the rule description of the target anomaly rule.

7. The method for detecting abnormal input of an intelligent agent based on a large model according to any one of claims 1 to 5, characterized in that: The method further comprises: If the abnormality detection result corresponding to the input content meets at least one of the following conditions, the input content is refused to be input into the first model associated with the agent: The abnormality detection result indicates that the input content is abnormal input content; The abnormality level corresponding to the input content in the abnormality detection result meets the preset level.

8. The method for detecting abnormal input of an intelligent agent based on a large model according to any one of claims 1 to 5, characterized in that: The service scope description information corresponding to the agent is obtained in the following way: Based on the fifth model, semantically parsing the agent's configuration file according to a preset service scope description information template is performed to obtain initial description information corresponding to the agent, wherein the preset service scope description information template includes a sub-template for generating description information corresponding to at least one of the agent's functional positioning, skill scope, and security restriction rules; The initial description information is formatted according to a preset format to obtain the service scope description information corresponding to the agent in the preset format.

9. The method for detecting abnormal input of an intelligent agent based on a large model according to any one of claims 1 to 5, characterized in that: The method further comprises: After the agent is updated, new service scope description information is generated according to the configuration file corresponding to the updated agent; Obtaining new input to the agent, The new input content and the new service scope description information are semantically understood based on the second largest model, and the anomaly detection result corresponding to the new input content is determined according to the semantic difference between the new input content and the new service scope description information.

10. A large-scale model-based intelligent agent abnormal input detection device, characterized in that: The device includes: An acquisition module, configured to acquire input content for an agent; wherein the agent is associated with a first large model, and the first large model is configured to output a model processing result based on the input content of the agent; A determination module is used to perform semantic understanding of the input content and the service scope description information corresponding to the intelligent agent based on the second largest model, and determine the anomaly detection result corresponding to the input content according to the semantic difference between the input content and the service scope description information; wherein, the service scope description information corresponding to the intelligent agent is determined based on the configuration file of the intelligent agent, and the configuration file is at least used to describe the definition of the service function provided by the intelligent agent.

11. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images