A large-model malicious instruction monitoring and early warning system and method

Abstract

The present invention discloses a large-model malicious instruction monitoring and early warning system and method, and relates to the field of network security. The present invention, by constructing a malicious instruction monitoring and early warning system that can learn autonomously and integrating it into the decision-making cycle of "observation-adjustment-decision-action", can monitor the input instructions of the large model in real time, and judge whether it has malicious intentions, quickly identify various forms of malicious instructions, and can resist malicious instruction attacks in time, and notify users or system administrators to take higher-level protective measures. In addition, the monitoring and early warning method proposed by the present invention also has strong adaptability and can flexibly adjust the monitoring strategy according to different application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of network security, and in particular to a large-model malicious instruction monitoring and early warning system and method. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] With the rapid development of artificial intelligence technology, especially the widespread application of large-scale language models in various fields, how to ensure the security and reliability of these models has become an urgent problem to be solved. Large models have powerful language generation and understanding capabilities through training with large amounts of data, but they also face the potential risk of malicious command attacks. Malicious command attacks refer to inducing large models to generate harmful content or perform unsafe operations by constructing specific inputs. Such attacks may not only cause system crashes and data leaks, but may also be used to carry out cyber attacks or spread false information, causing serious social and economic impacts.

[0004] In practical applications, large models are usually used as chatbots, intelligent assistants or content generators, etc., to interact directly with users, which significantly increases the risk of malicious command attacks. Existing malicious command monitoring technologies mainly rely on static rule matching and blacklist and whitelist mechanisms, which cannot effectively respond to dynamically changing attack patterns and increasingly complex attack techniques.

[0005] In addition, due to the generation capacity of large models, malicious command attacks often evade detection by disguising themselves as legitimate user input, which makes it difficult for existing security protection measures to play an effective role and fail to detect and respond to new malicious command attacks in a timely manner. This makes how to effectively monitor and warn of malicious command attacks faced by large models an important factor in ensuring the security of large models. Summary of the invention

[0006] The purpose of the present invention is to provide a large-model malicious instruction monitoring and early warning system and method in response to the problems existing in the prior art. By constructing a malicious instruction monitoring and early warning system that can learn autonomously and integrating it into the decision-making cycle of "observation-adjustment-decision-action", it is possible to monitor the input instructions of the large model in real time, and determine whether it has malicious intentions, quickly identify various forms of malicious instructions, and be able to resist malicious instruction attacks in a timely manner, and notify users or system administrators to take higher-level protective measures. In addition, the monitoring and early warning method proposed by the present invention also has strong adaptability and can flexibly adjust the monitoring strategy according to different application scenarios.

[0007] The technical solution of the present invention is as follows:

[0008] A large-model malicious instruction monitoring and early warning system, comprising:

[0009] Instruction safety multi-agent collaborative analysis framework and various tool modules;

[0010] The instruction security multi-agent collaborative analysis framework includes: a malicious instruction monitoring API call platform, a malicious instruction monitoring task scheduling model, and a malicious instruction monitoring API executor;

[0011] The malicious instruction monitoring API calling platform is responsible for API interface registration of each tool module and encapsulates each tool module into a corresponding intelligent entity;

[0012] The malicious instruction monitoring task scheduling model can identify the user's intention and generate a malicious instruction monitoring process task;

[0013] The malicious instruction monitoring API executor can call the corresponding intelligent agent to perform malicious instruction monitoring according to the malicious instruction monitoring process task, and feed back the monitoring results to the malicious instruction monitoring task scheduling model.

[0014] Furthermore, the tool module includes: a large model security knowledge base, a user behavior record library, a sensitive word monitoring and filtering tool, and a small instruction security monitoring model.

[0015] Furthermore, the malicious instruction monitoring task scheduling big model uses an open source general big model, which is responsible for identifying the user's intention based on the user's input instructions and the user's historical behavior records, and generating malicious instruction monitoring process tasks.

[0016] Furthermore, the instruction security monitoring model includes: an instruction adversarial analysis model, a malicious content monitoring model and a privacy theft monitoring model.

[0017] Furthermore, the training process of the instruction safety monitoring small model is as follows:

[0018] Step A: construct an initial sample of malicious instruction pairs of a large model; according to the three tasks of adversarial analysis, malicious content monitoring, and privacy theft monitoring, collect malicious instruction samples from known security vulnerabilities and attack cases, and analyze and classify them in combination with expert knowledge, and use an artificial construction method to generate N malicious instruction pairs, which constitute the initial sample of malicious instruction pairs; the malicious instruction pairs include: instructions and generated content;

[0019] Step B: Amplify the initial sample with malicious instructions; use two methods, sample synonym replacement and malicious instructions embedded in normal instructions, to achieve the amplification of the initial sample with malicious instructions;

[0020] Step C: Preparation of instruction security monitoring data set; the amplified malicious instruction samples are annotated in detail to indicate their categories and attack types; at the same time, normal instruction samples are collected and annotated for training the model to distinguish between positive and negative samples, and the preparation of four types of labeled data is completed, that is, the instruction security monitoring data set is obtained; the labeled data includes: instructions, generated content, labels; the categories of labels include: adversarial, malicious content, privacy theft and normal;

[0021] Step D: Training of instruction security monitoring model: select the BERT model architecture as the architecture of the instruction security monitoring model, and train the instruction adversarial analysis model, malicious content monitoring model, and privacy theft monitoring model respectively. Each model is a binary classification model.

[0022] Step E: Evaluate the capabilities of the instruction safety monitoring small model; conduct a comprehensive evaluation of the instruction safety monitoring small model on an independent test set, including testing of multiple performance indicators; at the same time, conduct batch input tests to evaluate the working efficiency of the instruction safety monitoring small model. If the current effect is not ideal, repeat steps A to E and repeat training.

[0023] The present invention also proposes a large-model malicious instruction monitoring and early warning method, based on the above-mentioned large-model malicious instruction monitoring and early warning system, comprising:

[0024] Step S1: The user's input command initiates a question to the target large model through the terminal service interface, and the input command is forwarded to the target large model and the command security multi-agent collaborative analysis framework through the application gateway;

[0025] Step S2: The malicious instruction monitoring task scheduling model in the instruction security multi-agent collaborative analysis framework identifies the user's intention based on the input instruction and the user's historical behavior records, and generates a malicious instruction monitoring process task;

[0026] Step S3: The malicious instruction monitoring API executor in the instruction security multi-agent collaborative analysis framework calls the corresponding agent to perform malicious instruction monitoring according to the malicious instruction monitoring process task, and feeds back the monitoring result to the malicious instruction monitoring task scheduling big model; if the monitoring result is a normal instruction, the input instruction is input into the target big model to generate the actual big model output expected by the user; if the monitoring result is a malicious instruction, the input instruction is filtered and input into the target big model, and the target big model returns a warning to the user.

[0027] Furthermore, the step S1 further includes:

[0028] Before input commands enter the application gateway, invalid or unauthorized questions are filtered out through firewall rules.

[0029] Furthermore, the step S2 comprises:

[0030] Step S21: collecting the user's current input command, other command data during this round of conversation, and the user's historical behavior records recorded in the user behavior record library, and pre-processing the data;

[0031] Step S22: Filling the preprocessed data into the pre-customized large model generation task prompt word template stored in the large model security knowledge base to generate a large model prompt word;

[0032] Step S23: Through the generated large model prompt words, the existing information stored in the large model security knowledge base is used to guide the malicious instruction monitoring task scheduling large model to understand and process new tasks, generate malicious instruction monitoring process tasks, and specify the type of monitoring to be performed, the priority of monitoring, and the expected execution path, including: specifying which small models need to be called, the scheduling order of tasks, and guidance on how to process and integrate various monitoring results.

[0033] Furthermore, the step S3 comprises:

[0034] Step S31: The malicious instruction monitoring task scheduling model generates necessary operation codes according to the calling API of the agent, and calls the malicious instruction monitoring API executor to execute each agent to ensure that the monitoring tasks can be accurately executed;

[0035] Step S32: The malicious instruction monitoring API executor collects the monitoring results of each intelligent agent and feeds them back to the malicious instruction monitoring task scheduling big model; if the monitoring result is a normal instruction, the input instruction is input into the target big model to generate the actual big model output expected by the user; if the monitoring result is a malicious instruction, the input instruction is filtered and input into the target big model, and the target big model returns a warning to the user. At the same time, it records the monitored malicious instruction category and the big model output that may be generated by the malicious instruction, and associates it with the user's historical behavior; if it is found that the user has requested malicious instructions multiple times in a short period of time, the user's account or IP is added to the system's blacklist, and the monitored warning is fed back to the administrator of the big model malicious instruction monitoring and warning system in the form of a system log.

[0036] Furthermore, the preprocessing in step S21 includes: truncating an overlong character string and selecting a recent conversation.

[0037] The present invention constructs a small model for command safety monitoring, establishes a multi-agent collaborative analysis framework for command safety, and realizes real-time online monitoring of user commands in a large model application system. Compared with the existing solutions, it has the following beneficial effects and advantages:

[0038] 1. By using a small model designed specifically for command security monitoring, the monitoring efficiency can be significantly improved while ensuring monitoring accuracy. The small model is optimized to quickly process a large amount of command data, thereby identifying potential security threats in a short time. Due to its efficient performance and accurate monitoring capabilities, it is particularly suitable for scenarios that require real-time online monitoring, ensuring rapid response to malicious commands and taking corresponding measures to ensure the security and stable operation of the system.

[0039] 2. The small model is combined with the centralized large model through the instruction safety multi-agent collaborative analysis framework. This not only optimizes the system's resource allocation and reduces the dependence on the central processing unit, but also further realizes the comprehensive analysis and result fusion of instruction safety by calling the general open source large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a block diagram of a large-model malicious command monitoring and early warning system;

[0041] Figure 2 The figure is a flow chart of a large-model malicious instruction monitoring and early warning method. DETAILED DESCRIPTION

[0042] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0043] First of all, it should be noted that the goal of large-model malicious command monitoring and early warning is to ensure the security and reliability of artificial intelligence systems and prevent malicious use and potential abuse.

[0044] Therefore, the present invention monitors and analyzes the user's command input and the model's output results in real time, identifies and warns of potential malicious behavior in advance, helps maintain the model's ethical and legal compliance, and enhances the public's trust in artificial intelligence technology.

[0045] The large-model malicious instruction monitoring and early warning system and method proposed in the present invention aims to achieve comprehensive security monitoring of complex instructions through a multi-agent system. Specifically, the present invention encapsulates multiple small models such as sensitive word filtering tools, instruction adversarial monitoring, malicious content monitoring, and privacy theft monitoring into independent agents, each of which is responsible for performing a specific type of security monitoring task. These agents communicate and exchange data through standardized interfaces to jointly complete the monitoring and early warning of malicious instructions.

[0046] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0047] Embodiment 1

[0048] This embodiment mainly proposes solutions to the large model instruction security problem, and specifically proposes solutions to the following problems:

[0049] 1. How to achieve real-time online monitoring of malicious instructions. In this embodiment, a small instruction security monitoring model is constructed for common instruction attack types in the large model, so as to greatly improve the monitoring efficiency while ensuring the monitoring accuracy.

[0050] 2. How to achieve accurate malicious command identification in complex user command situations. In this embodiment, by encapsulating the monitoring model or monitoring tool as an agent, a multi-agent collaborative analysis framework for command security is constructed to solve the multi-dimensional and multi-level analysis problem of malicious commands.

[0051] 3. How to achieve high scalability of monitoring and early warning methods for possible unknown malicious instruction attacks. In this embodiment, a new dedicated model or dedicated tool is encapsulated as an agent, and the new unknown malicious instruction attack monitoring can be completed through the instruction security multi-agent collaborative analysis framework.

[0052] For details, please refer to Figure 1 , a large-model malicious instruction monitoring and early warning system, comprising:

[0053] Instruction security multi-agent collaborative analysis framework and various tool modules; it should be noted that it is connected to the network through networking to form a large-model malicious instruction monitoring and early warning system;

[0054] The instruction security multi-agent collaborative analysis framework includes: a malicious instruction monitoring API call platform, a malicious instruction monitoring task scheduling model, and a malicious instruction monitoring API executor;

[0055] The malicious instruction monitoring API call platform is responsible for API interface registration of each tool module and encapsulates each tool module into a corresponding intelligent entity; this structure not only simplifies the API integration and operation process, but also enables the system to deploy and adapt to new malicious instruction categories more quickly;

[0056] The malicious instruction monitoring task scheduling model can identify the user's intention and generate a malicious instruction monitoring process task;

[0057] The malicious instruction monitoring API executor can call the corresponding intelligent agent to perform malicious instruction monitoring according to the malicious instruction monitoring process task, and feed back the monitoring results to the malicious instruction monitoring task scheduling model.

[0058] In this embodiment, specifically, the tool module includes: a large model security knowledge base, a user behavior record library, a sensitive word monitoring and filtering tool, and a small instruction security monitoring model;

[0059] The instruction security monitoring model includes: an instruction adversarial analysis model, a malicious content monitoring model and a privacy theft monitoring model;

[0060] It should be noted that the malicious instructions faced by large models can be mainly divided into the following categories:

[0061] First, adversarial attacks inject tiny perturbations into the model's input data, aiming to mislead the model into making incorrect predictions or decisions. This type of attack is particularly common in fields such as image recognition and natural language processing. Second, the input instructions themselves involve inappropriate or unreasonable topics, inducing the model to generate content that violates ethics and laws, such as violence, pornography, terrorism, crime, etc. There are also prompt leakage attacks, which may extract some prompts provided by the system by analyzing the output of the model, thereby obtaining sensitive information about the system itself or other users;

[0062] Therefore, this embodiment constructs three special small models to target the instruction adversarial attacks, malicious content generation, and privacy theft attacks that the target large model may suffer. They are: an instruction adversarial analysis small model, a malicious content monitoring small model, and a privacy theft monitoring small model. The use of special small models not only has higher detection accuracy, but more importantly, improves detection efficiency, and is used to provide real-time malicious instruction online detection services.

[0063] In this embodiment, specifically, the training process of the instruction safety monitoring small model is as follows:

[0064] Step A: Construct the initial sample of malicious instruction pairs of the large model; according to the three tasks of adversarial analysis, malicious content monitoring, and privacy theft monitoring, collect malicious instruction samples from known security vulnerabilities and attack cases, and analyze and classify them in combination with expert knowledge, and use the artificial construction method to generate 500 malicious instruction pairs. N malicious instruction pairs constitute the initial sample of malicious instruction pairs; the malicious instruction pairs include: instructions and generated content; it is necessary to ensure sufficient discrimination between instruction pairs, and the discrimination calculation can use the TF-IDF model to ensure the difference between samples;

[0065] Step B: Amplify the initial sample with malicious instructions; use two methods, sample synonym replacement and malicious instructions embedded in normal instructions, to achieve the amplification of the initial sample with malicious instructions;

[0066] It should be noted that sample synonym replacement is to replace the adjectives / verbs in malicious samples with similar semantics, and a synonym replacement table needs to be prepared; for example, replacing "fast" in the sample with "quickly" can generate a new text variant without changing the basic meaning of the original text;

[0067] Malicious instructions embedded in normal instructions are randomly inserted into a normal instruction, which is also considered to be a malicious instruction;

[0068] Step C: Preparation of instruction security monitoring data set; the amplified malicious instruction samples are annotated in detail to indicate their categories and attack types; at the same time, normal instruction samples are collected and annotated for training the model to distinguish between positive and negative samples, and the preparation of four types of labeled data is completed, that is, the instruction security monitoring data set is obtained; the labeled data includes: instructions, generated content, labels; the categories of labels include: adversarial, malicious content, privacy theft and normal;

[0069] Step D: Training of instruction security monitoring model: select the BERT model architecture as the architecture of the instruction security monitoring model, and train the instruction adversarial analysis model, malicious content monitoring model, and privacy theft monitoring model respectively. Each model is a binary classification model.

[0070] Step E: Evaluate the capabilities of the instruction security monitoring small model; conduct a comprehensive evaluation of the instruction security monitoring small model on an independent test set, including testing of multiple performance indicators; at the same time, conduct batch input tests to evaluate the working efficiency of the instruction security monitoring small model to ensure that it can respond quickly and accurately detect malicious instructions during actual deployment; if the current effect is not ideal, repeat steps A-E and repeat training.

[0071] In this embodiment, specifically, the malicious instruction monitoring task scheduling big model selects an open source general big model, such as: qwen2 or chatglm4, which is responsible for identifying the user's intention based on the user's input instructions and the user's historical behavior records, and generating a malicious instruction monitoring process task.

[0072] See also Figure 2 Based on the above-mentioned large-model malicious instruction monitoring and early warning system, this embodiment also proposes a large-model malicious instruction monitoring and early warning method, including:

[0073] Step S1: The user's input command initiates a question to the target large model through the terminal service interface. After the input command is filtered out by the firewall rules to remove invalid or unauthorized questions, it is forwarded to the target large model and the instruction security multi-agent collaborative analysis framework through the application gateway;

[0074] Step S2: The malicious instruction monitoring task scheduling model in the instruction security multi-agent collaborative analysis framework identifies the user's intention based on the input instruction and the user's historical behavior records, and generates malicious instruction monitoring process tasks, that is, further refines and splits the detection tasks, and finally executes them by each agent; that is, malicious instruction monitoring task planning is performed;

[0075] Step S3: The malicious instruction monitoring API executor in the instruction security multi-agent collaborative analysis framework calls the corresponding agent to perform malicious instruction monitoring according to the malicious instruction monitoring process task, and feeds back the monitoring result to the malicious instruction monitoring task scheduling big model; if the monitoring result is a normal instruction, the input instruction is input into the target big model to generate the actual big model output expected by the user; if the monitoring result is a malicious instruction, the input instruction is filtered and input into the target big model, and the target big model returns a warning to the user.

[0076] In this embodiment, specifically, step S2 includes:

[0077] Step S21: collect the user's current input command, other command data during this round of conversation, and the user's historical behavior records recorded in the user behavior record library, and pre-process the data (truncate overlong character strings and select the most recent conversation);

[0078] Step S22: Fill the preprocessed data into the pre-customized large model generation task prompt word template stored in the large model security knowledge base to generate a large model prompt word; that is, the large model prompt word is used to input into the malicious instruction monitoring task scheduling large model in the next step to generate a specific detection task process;

[0079] Step S23: Through the generated big model prompt words, the existing information stored in the big model security knowledge base is used to guide the malicious instruction monitoring task scheduling big model to understand and process new tasks, generate malicious instruction monitoring process tasks, and specify the type of monitoring to be performed, the priority of monitoring, and the expected execution path, including: specifying which small models need to be called, the scheduling order of tasks, and guidance on how to process and integrate various monitoring results;

[0080] It should be noted that the malicious instruction monitoring process task is implemented based on LangChain, which is an open source large-model task execution orchestration software used to build applications based on large language models (LLMs). LangChain is used to convert the malicious instruction monitoring process tasks output by the malicious instruction monitoring task scheduling large model into a series of executable task processes to ensure that each step can be executed smoothly and efficiently.

[0081] In this embodiment, specifically, step S3 includes:

[0082] Step S31: The malicious instruction monitoring task scheduling model generates necessary operation codes according to the calling API of the agent, and calls the malicious instruction monitoring API executor to execute each agent to ensure that the monitoring tasks can be accurately executed;

[0083] Step S32: The malicious instruction monitoring API executor collects the monitoring results of each intelligent agent and feeds them back to the malicious instruction monitoring task scheduling big model; if the monitoring result is a normal instruction, the input instruction is input into the target big model to generate the actual big model output expected by the user; if the monitoring result is a malicious instruction, the input instruction is filtered and input into the target big model, and the target big model returns a warning to the user. At the same time, the detected malicious instruction category and the big model output that may be generated by the malicious instruction are recorded and associated with the user's historical behavior; if it is found that the user has requested malicious instructions multiple times in a short period of time (usually set to 30 minutes), the user's account or IP is added to the system's blacklist, and the detected warning is fed back to the administrator of the big model malicious instruction monitoring and warning system in the form of a system log.

[0084] In this embodiment, it should be noted that the execution of each agent includes:

[0085] Call the sensitive word monitoring and filtering tool. This process is based on the rule engine, quickly discovers malicious information in the instructions, and feeds the monitoring results back to the malicious instruction monitoring task scheduling model, thus realizing the rapid monitoring of sensitive words.

[0086] Call the small model for instruction security monitoring to perform in-depth semantic recognition and detection on three types of attacks: instruction adversarial attacks, malicious content generation, and privacy theft attacks. Feedback the monitoring results to the large model for malicious instruction monitoring task scheduling, thus achieving comprehensive analysis of instruction security.

[0087] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0088] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A large-model malicious instruction monitoring and early warning method, characterized in that: A malicious command monitoring and early warning system based on a large model includes: Step S1: The user's input command initiates a question to the target large model through the terminal service interface, and the input command is forwarded to the target large model and the command security multi-agent collaborative analysis framework through the application gateway; Step S2: The malicious instruction monitoring task scheduling model in the instruction security multi-agent collaborative analysis framework identifies the user's intention based on the input instruction and the user's historical behavior records, and generates a malicious instruction monitoring process task; Step S3: The malicious instruction monitoring API executor in the instruction security multi-agent collaborative analysis framework calls the corresponding agent to monitor malicious instructions according to the malicious instruction monitoring process task, and feeds back the monitoring result to the malicious instruction monitoring task scheduling large model; if the monitoring result is a normal instruction, the input instruction is input into the target large model to generate the actual large model output expected by the user; if the monitoring result is a malicious instruction, the input instruction is filtered and input into the target large model, and the target large model returns a warning to the user; The step S2 comprises: Step S21: collecting the user's current input command, other command data during this round of conversation, and the user's historical behavior records recorded in the user behavior record library, and pre-processing the data; Step S22: Filling the preprocessed data into the pre-customized large model generation task prompt word template stored in the large model security knowledge base to generate a large model prompt word; Step S23: Through the generated large model prompt words, the existing information stored in the large model security knowledge base is used to learn the context to guide the malicious instruction monitoring task scheduling large model to understand and process new tasks, generate malicious instruction monitoring process tasks, and specify the type of monitoring to be performed, the priority of monitoring, and the expected execution path, including: specifying which small models need to be called, the scheduling order of tasks, and guidance on how to process and integrate various monitoring results; The step S3 comprises: Step S31: The malicious instruction monitoring task scheduling model generates an operation code according to the calling API of the agent, and calls the malicious instruction monitoring API executor to execute each agent to ensure that the monitoring task can be accurately executed; Step S32: The malicious instruction monitoring API executor collects the monitoring results of each intelligent agent and feeds them back to the malicious instruction monitoring task scheduling big model; if the monitoring result is a normal instruction, the input instruction is input into the target big model to generate the actual big model output expected by the user; if the monitoring result is a malicious instruction, the input instruction is filtered and input into the target big model, and the target big model returns a warning to the user. At the same time, the detected malicious instruction category and the big model output that may be generated by the malicious instruction are recorded and associated with the user's historical behavior; if it is found that the user has requested malicious instructions multiple times in a short period of time, the user's account or IP is added to the system's blacklist, and the detected warning is fed back to the administrator of the big model malicious instruction monitoring warning system in the form of a system log; A large-model malicious instruction monitoring and early warning system, comprising: Instruction safety multi-agent collaborative analysis framework and various tool modules; The instruction security multi-agent collaborative analysis framework includes: a malicious instruction monitoring API call platform, a malicious instruction monitoring task scheduling model, and a malicious instruction monitoring API executor; The malicious instruction monitoring API calling platform is responsible for API interface registration of each tool module and encapsulates each tool module into a corresponding intelligent entity; The malicious instruction monitoring task scheduling model can identify the user's intention and generate a malicious instruction monitoring process task; The malicious instruction monitoring API executor can call the corresponding intelligent agent to perform malicious instruction monitoring according to the malicious instruction monitoring process task, and feed back the monitoring results to the malicious instruction monitoring task scheduling model.

2. A large-model malicious instruction monitoring and early warning method according to claim 1, characterized in that: The step S1 further includes: Before input commands enter the application gateway, invalid or unauthorized questions are filtered out through firewall rules.

3. A large-model malicious instruction monitoring and early warning method according to claim 2, characterized in that: The preprocessing in step S21 includes: truncating an overlong character string and selecting the most recent conversation.

4. A large-model malicious instruction monitoring and early warning method according to claim 1, characterized in that: The tool module includes: a large model security knowledge base, a user behavior record library, a sensitive word monitoring and filtering tool, and a small instruction security monitoring model.

5. A large-model malicious instruction monitoring and early warning method according to claim 4, characterized in that: The malicious instruction monitoring task scheduling big model uses an open source general big model, which is responsible for identifying the user's intention based on the user's input instructions and the user's historical behavior records, and generating malicious instruction monitoring process tasks.

6. A large-model malicious instruction monitoring and early warning method according to claim 5, characterized in that: The instruction security monitoring model includes: an instruction adversarial analysis model, a malicious content monitoring model and a privacy theft monitoring model.

7. A large-model malicious instruction monitoring and early warning method according to claim 6, characterized in that: The training process of the instruction safety monitoring small model is as follows: Step A: Construct the initial sample of malicious instruction pairs of the large model; according to the three tasks of adversarial analysis, malicious content monitoring, and privacy theft monitoring, collect malicious instruction samples from known security vulnerabilities and attack cases, analyze and classify them in combination with expert knowledge, and use the artificial construction method to generate N malicious instruction pairs, which constitute the initial sample of malicious instruction pairs; The malicious instruction pair includes: an instruction and generated content; Step B: Amplify the initial sample with malicious instructions; use two methods, sample synonym replacement and malicious instructions embedded in normal instructions, to achieve the amplification of the initial sample with malicious instructions; Step C: Preparation of instruction security monitoring data set; the amplified malicious instruction samples are annotated in detail to indicate their categories and attack types; at the same time, normal instruction samples are collected and annotated for training the model to distinguish between positive and negative samples, and the preparation of four types of labeled data is completed, that is, the instruction security monitoring data set is obtained; the labeled data includes: instructions, generated content, labels; the categories of labels include: adversarial, malicious content, privacy theft and normal; Step D: Training of instruction security monitoring model: select the BERT model architecture as the architecture of the instruction security monitoring model, and train the instruction adversarial analysis model, malicious content monitoring model, and privacy theft monitoring model respectively. Each model is a binary classification model. Step E: Evaluate the capabilities of the instruction safety monitoring small model; conduct a comprehensive evaluation of the instruction safety monitoring small model on an independent test set, including testing of multiple performance indicators; at the same time, conduct batch input tests to evaluate the working efficiency of the instruction safety monitoring small model. If the current effect is not ideal, repeat steps A to E and repeat training.

Images