Safety protection method, device and equipment for large model agent and storage medium

Abstract

According to the embodiment of the invention, the invention provides a security protection method and device for a large model agent, equipment and a storage medium. The method comprises the following steps: acquiring a first processing request initiated to an intelligent agent based on a large model; determining intention similarity between the first processing request and a plurality of preset reference processing requests, wherein the plurality of reference processing requests are generated based on a service range provided by an intelligent agent; and outputting information allowing the intelligent agent to process the first processing request in response to the intention similarity meeting a predetermined condition, and outputting alarm information in response to the situation that the intention similarity does not meet a predetermined condition. Based on the mode, the security protection of the intelligent agent can be effectively improved, and the confidence and accuracy of information processing can be effectively improved.

Description

Technical Field

[0001] The exemplary embodiments of the present disclosure generally relate to the field of computers, and more particularly to a security protection method, apparatus, device, and computer-readable storage medium for a large-model intelligent agent. Background Art

[0002] With the development of computer technology, agent interaction technology has become an important research direction in the field of artificial intelligence. During the process of agent-user interaction, system prompts provide the agent with the initial framework and direction for interaction with the user, playing a vital role in the process of agent-user interaction. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for securing a large-model intelligent agent is provided. The method comprises: obtaining a first processing request initiated to an intelligent agent based on a large model; determining the intent similarity between the first processing request and a plurality of preset reference processing requests, wherein the plurality of reference processing requests are generated based on the scope of services provided by the intelligent agent; and, in response to the intent similarity satisfying a predetermined condition, outputting information allowing the intelligent agent to process the first processing request; and in response to the intent similarity not satisfying the predetermined condition, outputting a warning message.

[0004] In a second aspect of the present disclosure, a device for security protection of a large-model intelligent agent is provided. The device includes: an acquisition module configured to acquire a first processing request initiated to an intelligent agent based on a large model; a determination module configured to determine the intent similarity between the first processing request and a plurality of preset reference processing requests, wherein the plurality of reference processing requests are generated based on the scope of services provided by the intelligent agent; a first output module configured to output information allowing the intelligent agent to process the first processing request in response to the intent similarity satisfying a predetermined condition; and a second output module configured to output an alarm message in response to the intent similarity not satisfying the predetermined condition.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A flowchart of a security protection process for a large model agent according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A flowchart of a process for optimizing prompt information according to some embodiments of the present disclosure is shown;

[0012] Figure 4 An example flow chart illustrating information processing according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic structural block diagram of an apparatus for security protection of a large model intelligent agent according to certain embodiments of the present disclosure is shown;

[0014] Figure 6 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0016] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0018] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects shall comply with the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms them. Accordingly, when implementing the various embodiments of the present disclosure, the types, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method may vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0019] If this specification and the solutions in the examples involve the processing of personal information, such processing will be done only with a legitimate basis (such as with the consent of the subject of personal information or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of basic functions.

[0020] Traditionally, some solutions exist to address some security issues encountered during interactions between users and agents. For example, rule-based constraint solutions such as keyword matching and regular expression matching can be used to identify and intercept unsafe user input. Another example is the ability to determine the safety of user input and the agent's response based on classification models. Furthermore, when training models associated with the agent, security alignment data can be injected to fine-tune and reinforce the model, ensuring the security of the agent's responses.

[0021] However, these traditional solutions are ineffective in preventing jailbreaking attempts at the agent level. For example, if an agent is designed to perform data query services through a plug-in, users might be able to manipulate the agent to perform data modification operations through induced questions and answers. Furthermore, as the security capabilities of the model itself improve, users can use multiple rounds of induced questions to gradually deviate from the agent's original intent. Without the induced questions themselves being overtly malicious, traditional or content-based security solutions may become ineffective.

[0022] Embodiments of the present disclosure propose a security protection scheme for a large-model intelligent agent. According to the scheme, a first processing request initiated to an intelligent agent based on a large model is obtained; the intent similarity between the first processing request and multiple preset reference processing requests is determined, where the multiple reference processing requests are generated based on the service scope provided by the intelligent agent; and in response to the intent similarity satisfying a predetermined condition, information is output allowing the intelligent agent to process the first processing request; in response to the intent similarity not satisfying the predetermined condition, a warning message is output.

[0023] Based on this approach, the embodiments of the present disclosure can assist in determining whether a new processing request is within the service scope corresponding to the intelligent agent based on multiple reference processing requests generated by the service scope provided by the intelligent agent, and only allow the intelligent agent to process the new processing request when the new processing request is within the service scope corresponding to the intelligent agent, which can effectively improve the security protection of the intelligent agent and effectively improve the confidence and accuracy of information processing.

[0024] Sample Environment

[0025] Figure 1 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0026] In the example environment 100, the electronic device 110 may obtain a first processing request. Furthermore, the electronic device 110 may determine whether the external agent 120 is allowed to process the first processing request. If the agent 120 is allowed to process the first processing request, the agent 120 may process the first processing request and provide a response to the first processing request.

[0027] In some embodiments, the electronic device 110 can be any type of mobile terminal, fixed terminal or portable terminal with a display device, including a mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, PDA, portable game terminal, VR / AR device, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, game device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 can also support any type of interface for the target user (such as "wearable" circuit, etc.).

[0028] The electronic device 110 may also be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The electronic device 110 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0029] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0030] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0031] Example Process

[0032] Figure 2 FIG2 shows a flowchart of a security protection process 200 of a large model agent according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. Figure 1 Process 200 is described.

[0033] In block 210 , the electronic device 110 obtains a first processing request initiated to a large model-based agent.

[0034] In some embodiments, the big model is any appropriate machine learning model with a large number of parameters, which can be configured to process any appropriate complex task. For example, in the embodiment of the present disclosure, the big model can be a large language model (LLM), and the big model can also be a visual big model, an audio big model or a multimodal big model. The multimodal big model can process a variety of different types of data, such as multimodal data such as text, images, and audio. In some embodiments, the intelligent agent is a system or entity that can perceive the environment and make decisions or make decisions and take actions to achieve specific goals or tasks. In the industry, intelligent agents are also called agents. In some embodiments, the intelligent agent can use one or more big models to drive its interaction with the user, and the interaction can be any appropriate interaction. As an example, the intelligent agent can use the big model to generate response content for the processing request input by the user based on the processing request input by the user.

[0035] In some embodiments, the first processing request may be any appropriate type of request, such as a text type request, a voice type request, and the like.

[0036] In some embodiments, the first processing request may correspond to any appropriate intent associated with the user. For example, if the first processing request is "Please help me generate a travel guide for place A", then the intent corresponding to the first processing request may be the intent of generating a travel guide.

[0037] In block 220 , the electronic device 110 determines the similarity of intent between the first processing request and a plurality of preset reference processing requests, wherein the plurality of reference processing requests are generated based on a range of services provided by the agent.

[0038] In some embodiments, the multiple reference requests may be pre-generated hypothetical requests, that is, they may not be requests actually input by the user, but are reference requests that may be input by a simulated user.

[0039] In some embodiments, the multiple reference requests may be any appropriate requests, such as but not limited to positive reference requests and negative reference requests.

[0040] In some embodiments, the intent corresponding to the forward reference request may be within the service scope that the agent can provide. In some embodiments, the service scope represents a set of all services or functions that the agent can provide, which can also be represented as the boundary of the capabilities that the agent can provide.

[0041] In some embodiments, the scope of services provided by the agent is determined based on the agent's basic information and confirmed whitespace information associated with the agent. The agent's basic information may include, but is not limited to, at least one of the following: system prompt information used to configure the agent 120, plug-in description information associated with the agent 120, and knowledge base description information associated with the agent 120. The system prompt information may indicate the role, responsibilities, and scope of authority of the agent 120, etc., which may help define the functional boundaries and operational limitations of the agent 120. This system prompt information may also be referred to as a system prompt word. The plug-in description information may indicate the functions and uses corresponding to the plug-in associated with the agent 120. The plug-in associated with the agent 120 may be used to expand the capabilities of the agent 120, such as supporting the addition of new data processing functions or access to external services. The knowledge base description information may indicate information such as the structure and functions of the knowledge base that the agent 120 can access or rely on. This knowledge base may include any appropriate data that the agent 120 can process or access. The confirmed white sample information associated with the agent can be any appropriate information that can assist in understanding and helping to determine the scope of services provided by the agent, such as a confirmed white sample request intended to be within the scope of services provided by the agent and the response content associated with this white sample request, and other information.

[0042] In some embodiments, the forward reference request can be a request that meets the preset security constraints, or a business request associated with the business type supported and processed by the agent, etc. Requests that meet the predetermined security constraints can also be called requests that meet white intent (positive intent), where white intent represents normal, legal and non-malicious requests. In some embodiments, the predetermined security constraints can be any appropriate constraints, which can be used to constrain and ensure that the requests processed by the agent 120 do not cause security risks or violate security policies. For example, the predetermined security constraints can constrain the request to not include sensitive information, and can also constrain the request to comply with predetermined legal rules, and can also constrain the request to not contain information that deliberately guides the change of the system prompt words of the agent 120, etc. The business type can be any appropriate type, for example, the agent 120 can support the processing of business types related to knowledge popularization, but cannot process business types related to financial management, etc.

[0043] As an example, if the intelligent agent 120 can only provide security services related to knowledge popularization, then the forward reference request can be a safe request such as "Why does a rainbow appear after the rain" that corresponds to the intention of knowledge popularization.

[0044] In some embodiments, the intent corresponding to the negative reference request may not be within the scope of the services provided by the agent. In some embodiments, the negative reference request may be a request that does not meet predetermined security constraints and / or a business request that is not related to the business type supported and processed by the agent, etc. The request that does not meet the predetermined security constraints may be a request that meets a black intent (negative intent), where a black intent represents a request that may be abnormal, illegal, or may have malicious attack intent.

[0045] As an example, if the intelligent agent can only provide security services related to knowledge popularization, then the negative reference request can be "Please help me check the income of user A this year", which is not related to knowledge popularization and involves querying the private data of other users.

[0046] Taking the server generating the multiple reference processing requests as an example, the determination process of the multiple reference processing requests is described. Of course, the multiple reference processing requests can be generated by other devices (such as the electronic device 110) in addition to being generated by the server, and will not be described in detail here.

[0047] In some embodiments, the server may provide basic information and white sample information to the first model to generate the multiple reference processing requests. The first model may be any appropriate machine learning model (such as a generative model), which will not be described in detail here.

[0048] In some embodiments, positive reference requests and negative reference requests can be generated using the first model based on corresponding guide words, respectively. In some embodiments, the server can generate a positive reference request using the first guide word. The first guide word can indicate that the intention of the generated reference request is within the scope of services provided by the intelligent agent, and it can also specifically indicate that there is a process for generating a positive reference request and precautions, etc. In some embodiments, the server can generate a negative reference request using the second guide word. The second guide word can indicate that the intention of the generated reference request is not within the scope of services provided by the intelligent agent. As an example, the second guide word can indicate that the generated reference request is negative and there are certain risks, but it will not trigger a request for a predetermined security policy. In some embodiments, the second guide word can also specifically indicate the process for generating a negative reference request (such as the need to consider attack paths during the generation process, etc.) and precautions, etc.

[0049] In some embodiments, the system prompt information used to configure the agent 120 may define information such as the role, responsibilities, and authority scope of the agent 120. The prompt information may also be referred to as a prompt item or a prompt word.

[0050] The following takes the server determining the system prompt information as an example to illustrate the process of determining the system prompt information. It should be noted that the system prompt information can be generated by other devices (such as electronic device 110) in addition to being generated by the server, which will not be described in detail here.

[0051] To ensure that the system prompt information is not easily modified and that agent 120 can output a secure response, the server can perform security reinforcement on the initial prompt information corresponding to agent 120 and use the security-reinforced prompt information as the system prompt information corresponding to agent 120. The initial prompt information can include initial and unsecured information such as agent 120's role definition, functional scope, and processing permissions. The initial prompt information can also be called the initial system prompt.

[0052] In some embodiments, the server may obtain initial prompt information corresponding to agent 120. Furthermore, the server may determine risk assessment information corresponding to the initial prompt information. Specifically, the server may provide the initial prompt information to a trained model to obtain the risk assessment information. This trained model may be any appropriate machine learning model and is not described in detail herein.

[0053] In some embodiments, the risk assessment information may indicate at least one risk in the initial prompt information, which may cause the prompt information corresponding to the agent 120 to be tampered with, or may cause the response content of the agent 120 to be out of bounds, etc.

[0054] In some embodiments, the at least one risk may include but is not limited to a first risk, a second risk, a third risk, a fourth risk, etc. These risks may correspond to different levels of insecurity.

[0055] Regarding the first risk, it can be characterized by the fact that the initial prompt information does not set the corresponding authority scope of the intelligent agent 120, which may lead to the problem of authority abuse of the intelligent agent 120, such as causing the intelligent agent 120 to perform unauthorized operations. As an example, if the initial prompt information is "Please answer all questions of the user", then because the initial prompt information does not limit the scope of the answer of the intelligent agent 120, it may cause the output of the intelligent agent 120 to have a security risk. For another example, if the initial prompt information is "Intelligent agent 120 can help users manage their accounts", then because the initial prompt information does not specify which specific account management operations the intelligent agent 120 can perform, it may cause the intelligent agent 120 to be induced to perform unauthorized management operations, such as resetting passwords or modifying account settings.

[0056] The second risk may indicate that the initial prompt information does not define the processing logic of agent 120, which may cause the behavior of agent 120 to be unpredictable or bypassed. As an example, if the initial prompt information is "agent 120 should provide helpful suggestions," the vague definition of "helpful" in the initial prompt information may cause agent 120 to provide potentially unsafe suggestions without clear guidance.

[0057] The third risk is that the initial prompt includes information that supports reverse engineering, potentially allowing other users to exploit sensitive information. For example, if the initial prompt reads "Use #REFUND_CODE to initiate a refund," the user could exploit the sensitive information, "#REFUND_CODE," for performing unsafe operations.

[0058] The fourth risk may be that the initial prompt information lacks security constraints associated with the initial prompt information, which may result in the initial prompt word being tampered with or leaked. Security constraints are used to prevent the initial prompt word from being tampered with or leaked.

[0059] It should be noted that, for the initial prompt information, the corresponding risk in the initial prompt information may be one or more.

[0060] In some embodiments, the risk assessment information may indicate which information segments in the initial prompt information are at risk. That is, in addition to indicating what specific risks exist, the risk assessment information may also indicate the location information of the information segments containing this or these risks in the initial prompt information.

[0061] In order to improve the security of the interaction, in some embodiments, the server may provide risk assessment information, initial prompt information and security constraint information to the second model to obtain system prompt information. Specifically, the second model may be any appropriate machine learning model. The security constraint information is used to indicate the security measures that need to be applied to the initial prompt information to ensure that the operation of the agent 120 that applies the security-reinforced system prompt information complies with predetermined security standards. For example, the initial prompt information can be reinforced by clearly defining the authority boundaries of the agent 120, constraining the output format, and reverse protection to address the risks present in the initial prompt information.

[0062] It should be noted that the above-mentioned trained model and the second model can be the same model, that is, the model for determining the risk assessment information and the model for determining the system prompt information by adjusting the initial prompt information can be the same model. In order to ensure the accurate output of the model, in some embodiments, when the server uses this model to determine the risk assessment information, it can input first guidance information into the model so that the model can determine the risk assessment information. The first guidance information can indicate the specific requirements for determining the risk assessment information based on the initial prompt information. In other embodiments, when the server uses this model to determine the system prompt information by adjusting the initial prompt information, it can input second guidance information into the model so that the model can determine the system prompt information. The second guidance information can indicate the specific requirements for adjusting the initial prompt information.

[0063] In some embodiments, the security constraint information may indicate at least one security constraint to be applied to the initial prompt information. The at least one security constraint may include a first security constraint, a second security constraint, and a third security constraint. These security constraints are used to constrain content at different levels.

[0064] The first security constraint can be used to restrict the scope of permissions corresponding to agent 120, preventing it from performing unauthorized operations. For example, the first security constraint can instruct agent 120 to only perform authorized operations. For example, authorized operations can be any appropriate operation. For example, account query operations can be set as authorized operations, while data modification operations can be set as unauthorized operations.

[0065] The second security constraint may be used to constrain the format of the content output by the agent 120. For example, the second security constraint may be used to constrain the content output by the agent 120 to not include sensitive information, or to comply with legal regulations.

[0066] The third security constraint can be used to constrain the conversion of information in the initial prompt information that supports reverse deduction into information that does not support reverse deduction. As an example, the third security constraint can be used to encrypt or obfuscate information that supports reverse deduction (such as sensitive information).

[0067] In some embodiments, the at least one security constraint may be associated with the at least one risk. As an example, the third security constraint may be associated with the third risk. Specifically, in response to determining that the initial prompt information contains the third risk, the electronic device 110 may add the third security constraint to the initial prompt information to optimize the portion of the initial prompt information containing the third security constraint, thereby obtaining a security-hardened system prompt information.

[0068] In other embodiments, the first security constraint may be associated with the first risk. As an example, in response to determining that the initial prompt information contains the first risk, the electronic device 110 may add the first security constraint to the initial prompt information to optimize the portion of the initial prompt information containing the first risk, thereby obtaining a security-hardened system prompt information.

[0069] In other embodiments, the second security constraint may be associated with the second risk. For example, in response to determining that the initial prompt message presents the second risk, the electronic device 110 may add the second security constraint (e.g., binding corresponding legal provisions) to the initial prompt message to constrain the corresponding reply content of the agent 120.

[0070] In order to improve efficiency, in other embodiments, the server can also provide risk assessment information and initial prompt information to the second model to obtain system prompt information, that is, the second model has learned the security constraint information in advance, and can obtain the system prompt information directly based on the learned security constraint information when obtaining the risk assessment information and the initial prompt information.

[0071] Figure 3 An example flow chart of prompt information reinforcement according to some embodiments of the present disclosure is shown. Figure 3 Provide explanation.

[0072] In block 310 , the server performs a risk scan based on the initial prompt information 301 .

[0073] As an example, the server may perform a risk scan on the initial prompt information 301 and thereby obtain risk assessment information.

[0074] In some embodiments, the risk assessment information may indicate at least one risk in the initial prompt information, and this at least one risk may include at least one of the following: reverse deduction risk (third risk), constraint missing risk (first risk), logical fuzzy related risk (second risk), etc.

[0075] Regarding the first risk, it may indicate that the initial prompt information does not set the corresponding permission scope of agent 120, such as the issue of permission abuse of agent 120, which may cause agent 120 to perform unauthorized operations. Regarding the second risk, it may indicate that the initial prompt information does not define the processing logic of agent 120, which may make the behavior of agent 120 unpredictable or bypassable. Regarding the third risk, it may indicate that the initial prompt information includes information that supports reverse deduction, which may result in certain sensitive information being exploited by other users.

[0076] At block 320 , the server utilizes the model to optimize the prompt information.

[0077] In some embodiments, the server may provide the risk assessment information and the initial prompt information 301 to the model (eg, the second model) to optimize the initial prompt information 301 .

[0078] As an example, in response to the reverse deduction risk in the initial prompt information 301, the server may encrypt the sensitive information in the initial prompt information 301 based on the security constraint library to optimize the initial prompt information 301. The security constraint library includes multiple predetermined security constraints, which are used to indicate security measures that need to be applied to the initial prompt information to ensure that the operation of the intelligent agent 120 applying the security-hardened system prompt information complies with predetermined security standards to address the risks in the initial prompt information.

[0079] As another example, in response to the risk of missing constraints in the initial prompt information 301 , the server may add permission boundaries to the initial prompt information 301 based on the security constraint library to optimize the initial prompt information 301 .

[0080] As another example, in response to the existence of logical ambiguity risk in the initial prompt information 301 , the server may bind legal clauses to the initial prompt information 301 based on the security constraint library to optimize the initial prompt information 301 .

[0081] In block 330 , the server performs adversarial testing on the optimized prompt information.

[0082] As an example, the server may determine whether the reinforced prompt information can bypass the corresponding security constraints through synonym replacement or semantic deformation to determine whether the reinforced prompt information satisfies the predetermined verification constraints. As another example, the server may also determine whether the security constraints in the reinforced prompt information destroy the original functions of the intelligent agent 120 to determine whether the reinforced prompt information satisfies the predetermined verification constraints.

[0083] Furthermore, if the reinforced prompt information cannot bypass the corresponding security constraints through synonym replacement or semantic deformation, and it is determined that the security constraints in the reinforced prompt information do not destroy the original functions of the intelligent agent 120, the server can determine that the adversarial test has passed.

[0084] At block 340 , the server may deploy hardening prompt information in response to the adversarial test passing.

[0085] As an example, the server may apply the reinforced prompt information to the agent 120 to reconfigure the agent 120. Furthermore, the server may generate multiple reference processing requests based on the reinforced prompt information.

[0086] Back to the process of generating these multiple reference processing requests,

[0087] Furthermore, the electronic device 110 may determine the similarity of the intention between the first processing request and the preset multiple reference processing requests (including positive reference requests and / or negative reference requests) based on a comparison between the first processing request and the multiple reference processing requests associated with the agent 120.

[0088] Taking the multiple reference processing requests including positive reference requests and negative reference requests as an example, the electronic device 110 may respectively determine a first similarity between the first processing request and the positive reference request and a second similarity between the first processing request and the negative reference request.

[0089] In some embodiments, the higher the first similarity between the first processing request and the positive reference request, the more likely it is that the intent of the first processing request is within the service scope provided by the agent, and the more likely it is that the agent 120 can support processing the first processing request. The higher the second similarity between the first processing request and the negative reference request, the more likely it is that the intent of the first processing request is not within the service scope provided by the agent, and the less likely it is that the agent 120 can support processing the first processing request.

[0090] Specifically, the electronic device 110 can determine the first similarity between the first processing request and the forward reference request based on the comparison between the forward reference request and the first processing request. The forward reference request can be one or more. As an example, if there are multiple forward reference requests, the electronic device 110 can respectively determine the first candidate similarities between these multiple forward reference requests and the first processing request. Further, the electronic device 110 can determine the highest value corresponding to the first candidate similarity as the first similarity. For example, if there are 3 forward reference requests, and the first candidate similarities between these 3 forward reference requests and the first processing request are 0.7, 0.6, and 0.5 respectively, the electronic device 110 can determine 0.7 as the first similarity.

[0091] Furthermore, the electronic device 110 may determine the second similarity between the first processing request and the negative reference request based on the comparison between the negative reference request and the first processing request. The negative reference request may be one or more. As an example, if there are multiple negative reference requests, the electronic device 110 may respectively determine the second candidate similarities between these multiple negative reference requests and the first processing request. Furthermore, the electronic device 110 may determine the sum value corresponding to the second candidate similarities as the second similarity. For example, if there are two negative reference requests, and the second candidate similarities between these two negative reference requests and the first processing request are 0.3 and 0.4 respectively, the electronic device 110 may determine the second similarity to be 0.7.

[0092] It should be noted that, in order to avoid data overflow, the electronic device 110 may set the second similarity to be less than a threshold, that is, the sum value corresponding to the second candidate similarities may be less than the threshold, and the threshold may be 1. When the sum value corresponding to the multiple second candidate similarities is greater than the threshold, the multiple second candidate similarities may be normalized, and the sum value of the normalized similarities may be determined as the second similarity.

[0093] In some embodiments, the electronic device 110 may directly determine the intention similarity between the first processing request and the multiple reference processing requests based on the first similarity and the second similarity.

[0094] For example, the electronic device 110 may determine the intent similarity based on the following formula:

[0095] y1=α×x1-β×x2

[0096] Among them, y1 is the intention similarity, x1 is the first similarity, x2 is the second similarity, α and β are predetermined coefficients, which can be set according to needs.

[0097] In other embodiments, the electronic device 110 determines the business relevance between the first processing request and the business scenario description information of the agent 120. As an example, the electronic device 110 may determine the business relevance based on the comparison of keywords in the first processing request with keywords in the business scenario description information. As another example, the electronic device 110 may determine the business relevance based on the semantic content corresponding to the first processing request and match it with the business scenario description information. For example, if the first processing request involves "returns and exchanges", and the business scenario description information of the agent 120 includes description information related to after-sales service, the electronic device 110 may determine that the business relevance between the first processing request and the business scenario description information of the agent is 0.9.

[0098] Furthermore, the electronic device 110 may determine the intention similarity between the first processing request and the plurality of reference processing requests based on the first similarity, the second similarity, and the business relevance.

[0099] For example, the electronic device 110 may determine the intent similarity based on the following formula:

[0100] y2=α×x1+β×x2-γ×x3

[0101] Where y2 is the matching degree, x1 is the first similarity, x2 is the business relevance, x3 is the third similarity, α, β, and γ are predetermined coefficients that can be set as needed. As an example, α+β=1 can be set (usually α>β, for example, α=0.7, β=0.3), and γ∈[0,0.3] (for example, it can be set to 0.2).

[0102] In some embodiments, when comparing the first processing request with the reference processing request, the electronic device 110 may compare the first intention corresponding to the first processing request and the second intention of the reference processing request. Further, the electronic device 110 may determine the similarity between the first intention and the second intention. For example, if the first processing request is "I need to export order data and unlock administrator privileges", and the similarity between the intent corresponding to the first processing request and the intent corresponding to the "data query" in the positive reference request is 0.8, and the similarity between the intent corresponding to the intent corresponding to the "authority application" is 0.6, and the similarity between the intent corresponding to the processing request and the intent corresponding to the "data export risk" of the negative reference request is 0.4, and the similarity between the intent corresponding to the intent corresponding to the "authority escalation risk" is 0.5, and the business relevance is 0.2, then the intent similarity between the first processing request and the reference processing request is y2 = 0.7 × 0.8 + 0.3 × 0.2 - 0.2 × 0.9 = 0.44.

[0103] In block 230 , the electronic device 110 outputs information allowing the agent to process the first processing request in response to the intention similarity satisfying a predetermined condition.

[0104] In some embodiments, the predetermined condition may be any appropriate condition, such as whether the value is greater than a predetermined threshold, etc. The predetermined threshold may be any appropriate threshold, such as 0.85, 0.9, etc., which will not be described in detail here.

[0105] Specifically, electronic device 110 can compare the intent similarity with a predetermined threshold to determine whether the intent similarity is greater than the predetermined threshold. In some embodiments, electronic device 110 can output information allowing the agent to process the first processing request in response to the intent similarity being greater than the predetermined threshold. Further, agent 120 can process the first processing request to generate a response result for the first processing request.

[0106] In block 240 , the electronic device 110 outputs warning information in response to the intention similarity not satisfying a predetermined condition.

[0107] In some embodiments, electronic device 110 may output a warning message in response to the intent similarity being less than or equal to a predetermined threshold. In some embodiments, the warning message indicates that agent 120 is not allowed to process the first processing request and instead directly rejects the response. Furthermore, electronic device 110 may also record a log so that the user can review relevant information related to agent 120's rejection of the response.

[0108] Figure 4 An example flow chart of information processing according to some embodiments of the present disclosure is shown. Figure 4 Provide explanation.

[0109] In block 401 , the electronic device 110 obtains user input.

[0110] As an example, the user's input can be any appropriate processing request. In some embodiments, the processing request can be any appropriate type of request input by the user, such as a text type request, a voice type request, etc.

[0111] As an example, the processing request may be obtained via a conversational interface between the user and the agent 120 .

[0112] In block 402 , the electronic device 110 pre-processes the user's input.

[0113] As an example, the preprocessing may be any appropriate processing, such as word segmentation, denoising, etc.

[0114] In some embodiments, the pre-processing operation can effectively improve the accuracy of the semantic parsing performed in the subsequent block 403 .

[0115] In block 403 , the electronic device 110 performs semantic parsing on the pre-processed user input.

[0116] As an example, the electronic device 110 may perform semantic analysis on the pre-processed user input to obtain the intention corresponding to the user input.

[0117] As an example, the intent may support the business information to be processed corresponding to the user, and may also indicate whether the user corresponds to a black intent or a white intent.

[0118] In block 404 , the server generates an intent sample based on the confirmed white sample information and the basic information of the agent 120 using the model.

[0119] In some embodiments, the intent samples are positive reference requests and / or negative reference requests.

[0120] As an example, the basic information of the agent 120 may include system prompt information for configuring the agent 120 , plug-in description information associated with the agent 120 , and knowledge base description information associated with the agent 120 .

[0121] In some embodiments, the system prompt information may indicate the role, responsibilities, and scope of authority of the agent 120, etc., which may assist in understanding the functional boundaries and operational limitations of the agent 120. The plug-in description information may indicate the functions and uses corresponding to the plug-ins associated with the agent 120. The plug-ins associated with the agent 120 may be used to expand the capabilities of the agent 120, which may support adding new data processing functions or accessing external services, etc. The knowledge base description information may indicate information such as the structure and functions of the knowledge base that the agent 120 can access or rely on. This knowledge base may include any appropriate data that the agent 120 can process or access.

[0122] In some embodiments, the confirmed white sample information associated with the agent can be any appropriate information that can assist in understanding and helping to determine the scope of services provided by the agent, such as a confirmed white sample request intended to be within the scope of services provided by the agent and the response content associated with the white sample request, and other information.

[0123] In block 405 , the server adds the intent sample to the knowledge base.

[0124] In some embodiments, the server may determine to add intent samples to the knowledge base to facilitate determining the similarity between the processing request input by the user and the intent corresponding to these intent samples.

[0125] In block 406 , the electronic device 110 may perform similarity calculation using the speech analysis result of the user input and the intent sample.

[0126] As an example, the electronic device 110 may determine a first similarity between the intent corresponding to the processing request and the intent corresponding to the positive intent sample (positive reference request). The electronic device 110 may determine a second similarity between the intent corresponding to the processing request and the intent corresponding to the negative intent sample (negative reference request). The electronic device 110 may determine the business relevance between the processing request and the business scenario description information of the agent 120.

[0127] Furthermore, the electronic device 110 may determine the intent similarity between the first processing request and the intent sample based on the first similarity, the second similarity, and the business relevance.

[0128] Furthermore, electronic device 110 may determine whether the intent similarity is greater than a threshold. As an example, electronic device 110 may perform the operation of block 407 in response to determining that the intent similarity is greater than the threshold. As another example, electronic device 110 may perform the operation of block 408 in response to determining that the intent similarity is less than or equal to the threshold.

[0129] In block 407 , the agent 120 may perform corresponding operations based on the user's input.

[0130] As an example, the intelligent agent 120 may generate response content corresponding to the user's input based on the user's input, and feed it back to the user.

[0131] In block 408 , the electronic device 110 may block execution of a corresponding operation based on the user's input and issue an alarm.

[0132] As an example, the electronic device 110 may output an alarm message and block the agent from generating response content corresponding to the user's input based on the user's input.

[0133] In some embodiments, the warning information may be any appropriate information, such as text-type information, voice-type information, and the like.

[0134] Based on this approach, the embodiments of the present disclosure can generate multiple reference processing requests based on the service scope provided by the intelligent agent to assist in determining whether a new processing request is within the service scope corresponding to the intelligent agent, and only allow the intelligent agent to process the new processing request when the new processing request is within the service scope corresponding to the intelligent agent, which can effectively improve the security protection of the intelligent agent and effectively improve the confidence and accuracy of information processing.

[0135] Example devices and equipment

[0136] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an apparatus 500 for security protection of a large model intelligent agent according to certain embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in the electronic device 110 discussed above. The various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0137] like Figure 5 As shown, the device 500 includes an acquisition module 510, configured to acquire a first processing request initiated to an intelligent agent based on a large model; a determination module 520, configured to determine the intention similarity between the first processing request and a preset plurality of reference processing requests, wherein the plurality of reference processing requests are generated based on the scope of services provided by the intelligent agent; a first output module 530, configured to output information allowing the intelligent agent to process the first processing request in response to the intention similarity satisfying a predetermined condition; and a second output module 540, configured to output an alarm message in response to the intention similarity not satisfying the predetermined condition.

[0138] In some embodiments, the multiple reference processing requests include at least one of the following: a positive reference request, wherein the intention of the positive reference request is within the scope of the service provided by the intelligent agent; a negative reference request, wherein the intention of the negative reference request is not within the scope of the service provided by the intelligent agent.

[0139] In some embodiments, the determination module 520 is further configured to: determine a first similarity between the first processing request and the positive reference request based on a comparison between the positive reference request and the first processing request; determine a second similarity between the first processing request and the negative reference request based on a comparison between the negative reference request and the first processing request; and determine the intention similarity between the first processing request and multiple reference processing requests based on the first similarity and the second similarity.

[0140] In some embodiments, the determination module 520 is further configured to: determine the business relevance between the first processing request and the business scenario description information of the intelligent entity; and determine the intention similarity between the first processing request and multiple reference processing requests based on the first similarity, the second similarity and the business relevance.

[0141] In some embodiments, the scope of services provided by the agent is determined based on the basic information of the agent and the confirmed white sample information associated with the agent.

[0142] In some embodiments, the plurality of reference processing requests are generated based on providing the base information and the white sample information to the first model to generate the plurality of reference processing requests.

[0143] In some embodiments, the basic information includes at least one of the following: system prompt information for configuring the agent; plug-in description information associated with the agent; and knowledge base description information associated with the agent.

[0144] In some embodiments, the system prompt information is determined based on the following process: obtaining initial prompt information corresponding to the intelligent agent; determining risk assessment information corresponding to the initial prompt information, the risk assessment information indicating at least one risk present in the initial prompt information; and adjusting the initial prompt information based on the risk assessment information to determine the system prompt information.

[0145] In some embodiments, at least one risk includes at least one of the following: a first risk, indicating that the initial prompt information does not define the authority scope of the intelligent entity; a second risk, indicating that the initial prompt information does not define the processing logic of the intelligent entity; a third risk, indicating that the initial prompt information includes information supporting reverse deduction; a fourth risk, indicating that the initial prompt information lacks security constraints associated with the initial prompt information.

[0146] In some embodiments, adjusting the initial prompt information based on the risk assessment information to determine the system prompt information includes: providing the risk assessment information, the initial prompt information and the security constraint information to the second model to obtain the system prompt information, the security constraint information indicating at least one security constraint to be applied to the initial prompt information. The units included in the device 500 can be implemented in various ways, including software, hardware, firmware or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0147] Figure 6 1 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 6 The illustrated electronic device 600 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to implement Figure 1An electronic device 110 is shown.

[0148] like Figure 6 As shown, electronic device 600 is in the form of a general electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 600.

[0149] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 600.

[0150] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 6 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0151] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0152] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0153] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0154] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0156] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0157] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0158] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A security protection method for a large-scale intelligent agent, comprising: Obtaining a first processing request initiated to the large model-based agent; determining a similarity in intent between the first processing request and a plurality of preset reference processing requests, wherein the plurality of reference processing requests are generated based on a range of services provided by the agent; as well as In response to the intention similarity satisfying a predetermined condition, outputting information allowing the agent to process the first processing request; In response to the intention similarity not satisfying a predetermined condition, outputting a warning message.

2. The method according to claim 1, wherein the plurality of reference processing requests comprises at least one of the following: a forward reference request, wherein the forward reference request is intended to be within the scope of services provided by the agent; A negative reference request, wherein the intention of the negative reference request is not within the scope of the service provided by the agent.

3. The method according to claim 2, wherein determining the similarity between the first processing request and a plurality of preset reference processing requests comprises: determining a first similarity between the first processing request and the forward reference request based on a comparison between the forward reference request and the first processing request; determining a second similarity between the first processing request and the negative reference request based on a comparison between the negative reference request and the first processing request; as well as Based on the first similarity and the second similarity, the intention similarity between the first processing request and the plurality of reference processing requests is determined.

4. The method of claim 3 , wherein determining the similarity of the intent between the first processing request and the plurality of reference processing requests comprises: Determining the business relevance between the first processing request and the business scenario description information of the agent; as well as The intention similarity between the first processing request and the plurality of reference processing requests is determined based on the first similarity, the second similarity, and the business relevance.

5. The method according to claim 1, wherein the service scope provided by the agent is determined based on basic information of the agent and confirmed white sample information associated with the agent.

6. The method of claim 5, wherein the plurality of reference processing requests are generated based on the following process: The basic information and the white sample information are provided to a first model to generate the plurality of reference processing requests.

7. The method according to claim 5, wherein the basic information includes at least one of the following: System prompt information for configuring the agent; Plug-in description information associated with the agent; The knowledge base description information associated with the agent.

8. The method according to claim 7, wherein the system prompt information is determined based on the following process: Obtaining initial prompt information corresponding to the agent; determining risk assessment information corresponding to the initial prompt information, the risk assessment information indicating at least one risk present in the initial prompt information; and Based on the risk assessment information, the initial prompt information is adjusted to determine the system prompt information.

9. The method of claim 8, wherein the at least one risk comprises at least one of the following: The first risk indicates that the initial prompt information does not define the authority scope of the agent; The second risk indicates that the initial prompt information does not define the processing logic of the agent; The third risk indicates that the initial prompt information includes information supporting reverse deduction; The fourth risk indicates that the initial prompt information lacks a security constraint associated with the initial prompt information.

10. The method according to claim 8, wherein adjusting the initial prompt information based on the risk assessment information to determine the system prompt information comprises: The risk assessment information, the initial prompt information, and security constraint information are provided to a second model to obtain the system prompt information, wherein the security constraint information indicates at least one security constraint to be applied to the initial prompt information.

11. A device for security protection of a large model intelligent agent, comprising: An acquisition module is configured to acquire a first processing request initiated to an agent based on a large model; a determination module configured to determine the similarity of intent between the first processing request and a plurality of preset reference processing requests, wherein the plurality of reference processing requests are generated based on the scope of services provided by the agent; a first output module configured to output information allowing the agent to process the first processing request in response to the intention similarity satisfying a predetermined condition; as well as The second output module is configured to output warning information in response to the intention similarity not meeting a predetermined condition.

12. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 10 when executed by a processor.

Images