A self-supervised large-model malicious instruction detection method and system

Through self-supervised learning and deep neural network combined with rules engine, using unlabeled data and user historical access records, a malicious instruction detection system was built, which solved the problem of insufficient malicious instruction detection capabilities in the existing technology, and achieved more efficient malicious instruction Detection and defense.

CN118784377BActive Publication Date: 2024-11-15CHINA ELECTRONICS TECH CYBER SECURITY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411267797.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-15
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

The prior art has limited detection capabilities when defending against large-scale malicious instruction attacks, especially because it relies on a large amount of labeled data and is difficult to cover all possible attack forms, resulting in high data labeling costs and difficult to effectively detect new types of malicious instruction attacks.

Method used

Using self-supervised learning method, through unlabeled potential information in instruction data, combined with instruction feature extraction, deep neural malicious instruction detection model and rule engine, samples are generated and screened, and a malicious instruction detection system based on fusion features is built to improve detection performance and Generalization ability.

Benefits of technology

Without significantly increasing the cost of data labeling, the accuracy and robustness of malicious instruction detection are improved, effective detection and defense of malicious instructions are achieved, and timely resistance to malicious instruction attacks is supported.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118784377B_ABST
    Figure CN118784377B_ABST
Patent Text Reader

Abstract

The present invention discloses a self-supervised large-model malicious instruction detection method and system, which relates to the field of network security. The large-model malicious instruction detection system is composed of a user terminal, a firewall, an instruction feature extractor, a malicious instruction detector, a target large model, an instruction sample generator and an instruction filter connected through a network, and a detection method is proposed based on this. The present invention, by utilizing the potential information in unlabeled instruction data, can effectively improve the malicious instruction detection performance and generalization ability of the model without significantly increasing the data annotation cost, thereby constructing a more robust malicious instruction detection model and supporting timely defense against malicious instruction attacks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of network security, and in particular to a self-supervised large-model malicious instruction detection method and system. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] With the rapid development of artificial intelligence and deep learning technologies, large-scale pre-trained models have achieved remarkable results in natural language processing, computer vision and other fields. However, with the widespread application of large models, their security issues have gradually attracted widespread attention from academia and industry. Malicious instruction attacks, as a new type of attack method, can induce the model to produce erroneous or harmful outputs by inputting carefully designed malicious instructions into the large model, thus posing a serious threat to the security and reliability of the system. This attack method can not only interfere with the reasoning results of the model, but may also be used to spread harmful information, create public opinion, etc., which has great potential harm.

[0004] Traditional model security research mainly focuses on defending against adversarial sample attacks and data poisoning attacks, while research on defense against malicious command attacks is relatively rare. The concealment and complexity of malicious command attacks make it difficult for existing defense methods to effectively detect and prevent them. At present, large-model malicious command detection mostly adopts methods based on rule engines and supervised learning, and has limited defense capabilities against new malicious command attacks. Moreover, supervised learning relies on a large amount of labeled data, which has high data labeling costs and is difficult to cover all possible forms of attacks. In this context, self-supervised learning, as a learning method that combines a small amount of labeled data with a large amount of unlabeled data, shows its unique advantages in malicious command detection. Summary of the invention

[0005] The purpose of the present invention is to provide a self-supervised large-model malicious instruction detection method and system to address the problems existing in the prior art. By utilizing the potential information in unlabeled instruction data, the malicious instruction detection performance and generalization ability of the model can be effectively improved without significantly increasing the data annotation cost, thereby constructing a more robust malicious instruction detection model to support timely defense against malicious instruction attacks.

[0006] The technical solution of the present invention is as follows:

[0007] A self-supervised large-model malicious instruction detection system, comprising:

[0008] User terminal, firewall, instruction feature extractor, malicious instruction detector and target large model;

[0009] The user terminal asks questions to the target large model by requesting the large model application service port, and filters out invalid or unauthorized access through firewall rules;

[0010] The instruction feature extractor is used to extract instruction features of the input text of the question;

[0011] The malicious instruction detector performs malicious instruction detection based on the instruction features extracted by the instruction feature extractor, and transmits the detection results to the target big model, which generates the actual big model output expected by the user or malicious user warning information.

[0012] Furthermore, it also includes:

[0013] Instruction sample generator and instruction filter;

[0014] The instruction feature extractor can also extract instruction features from the user history access record library, and the instruction features extracted by the instruction feature extractor are stored in the instruction feature library;

[0015] The instruction sample generator can automatically generate specific malicious instructions or normal instructions with similar semantics but different forms based on the rule engine and instruction feature library;

[0016] The instruction filter is used to filter the instructions generated by the instruction sample generator, and add the filtering results to the instruction sample library for training and updating the malicious instruction detector.

[0017] Furthermore, the malicious instruction detector is internally loaded with a deep neural malicious instruction detection model based on fusion features, and the deep neural malicious instruction detection model includes: a feature fusion network, a malicious instruction discrimination network, and a model update mechanism;

[0018] The feature fusion network can fuse instruction sample features of different dimensions;

[0019] The malicious instruction identification network can classify the fused features into multiple categories, identify whether it is a malicious instruction, and output the category of the malicious instruction;

[0020] The model update mechanism can realize incremental updates of the model without affecting business operations.

[0021] Furthermore, the instruction sample generator internally maintains an instruction generation module based on a rule engine; two types of instruction generation rule templates are defined in the rule engine, one is malicious instruction rule sample generation, the purpose of which is to generate malicious instructions with specific categories through normal user instructions; the other is neutral rule sample generation, the purpose of which is to generate a batch of instructions with the same functions as the original instructions but different character forms without changing the original instruction semantics, thereby expanding the number of instruction samples.

[0022] Furthermore, the instruction filter has an open source universal big model, and instruction filtering is performed through the open source universal big model.

[0023] Furthermore, the extraction method of the instruction feature extractor is as follows:

[0024] First, split the instruction to obtain instruction fragments;

[0025] Then preprocessing the instruction fragments; the preprocessing includes: removing noise, removing illegal characters, removing duplication, and standardizing;

[0026] Finally, string features and word features are extracted for the preprocessed instruction fragments, and natural language processing tools are used to complete instruction feature extraction.

[0027] Furthermore, the generation process of the instruction sample generator is as follows:

[0028] Get instruction snippets from the instruction signature library and the corresponding eigenvectors;

[0029] Randomly select instruction generation rules according to the category requirements of instruction sample generation , generate rules according to instructions Preconditions for execution, select instruction fragments The target of a certain behavior rule execution If there is no matching target, the instruction generation rule is changed. If all instruction rules are executed and no target is found, the next instruction fragment is changed. Continue execution;

[0030] On target , execute instruction generation rules , generate malicious instruction samples of corresponding categories Or normal instruction sample ; thereby generating a malicious instruction sample set or a normal instruction sample set;

[0031] The malicious instruction sample set and the normal instruction sample set are labeled according to the sample category and added to the instruction security detection sample library.

[0032] Furthermore, the screening process of the instruction filter is as follows:

[0033] Obtain the constructed malicious instruction samples from the instruction security detection sample library , Normal instruction sample ;

[0034] The malicious command sample , Normal instruction sample Input the prompt word template into the open source general large model to collect the detection results of the open source general large model;

[0035] For input is a normal instruction sample , assuming that the detection results of the open source general large model mostly do not detect maliciousness, then the normal instruction samples generated For correctly labeled samples, add them to the instruction sample library; otherwise, discard them;

[0036] The input is a malicious instruction sample ; Assuming that the detection results of the open source general large model mostly detect specific malicious categories, the generated malicious instruction samples For correctly labeled samples, add them to the instruction sample library; otherwise, discard them.

[0037] Furthermore, the feature fusion network utilizes three types of features: character level, word level, and sentence level, and uses an embedding method to convert text into a vector form; define Represents three feature fusion networks; It is each layer of the feature fusion network; is the input vector of the feature fusion network, including: character-level embedding feature vector, word-level embedding feature vector and sentence embedding feature vector; Represents input to The vector of layers; is a hyperparameter; yes The output of the layer; express The weight of the layer; yes Layer offset; neural network activation function Using a variant of the linear rectification function Leaky ReLU, the forward transfer formula of the feature fusion network is:

[0038]

[0039]

[0040]

[0041] in:

[0042] Represents the 0th layer output of the feature fusion network;

[0043] Represents the activation function The output of , that is, when it is greater than zero, output the value of the original function, otherwise, output 0;

[0044] definition It is a malicious instruction to identify each layer of the network. Represents input to The vector of layers, yes The output of the layer, express The weight of the layer, yes Layer offset, neural network activation function The variant linear rectification function Leaky ReLU is also used. is the output function of the malicious instruction discrimination network. Since it is a multi-classification problem, the softmax function is used; the forward transfer formula of the malicious instruction discrimination network is:

[0045]

[0046]

[0047]

[0048]

[0049] in:

[0050] Represents the input vector of the multi-layer neural network;

[0051] Represents the linear combination result of the first layer, Similarly;

[0052] represents the output of the 4th layer, with the softmax function applied;

[0053] represents the first The index of the neuron;

[0054] represents a natural constant;

[0055] Indicates the malicious command identification network The linear combination result of neurons;

[0056] Represents the index of a neuron in the output layer;

[0057] represents the total number of neurons in the output layer;

[0058] Indicates The linear combination result of neurons;

[0059] use represents the label of the training data, represents the output label of the malicious instruction identification network; then use Find the index of the maximum value of the parameter as the output of the malicious instruction detector; the cross entropy loss function used in the training process of the deep neural malicious instruction detection model based on fusion features is as follows:

[0060]

[0061] in:

[0062] Represents the cross entropy loss function, which is used to evaluate the match between the predicted output of the classification model and the true label;

[0063] Represents the index of the data sample, that is, represents each malicious instruction sample;

[0064] Represents the index of the maximum value, which is used to select the category with the highest predicted probability as the final output of the model.

[0065] A self-supervised large-model malicious instruction detection method, based on the above-mentioned self-supervised large-model malicious instruction detection system, comprises:

[0066] Step S1: The user terminal asks questions to the target large model by requesting the large model application service port, and filters out invalid or unauthorized access through firewall rules;

[0067] Step S2: using an instruction feature extractor to extract instruction features of the input text of the question in real time;

[0068] Step S3: Input the command features extracted in step S2 into the malicious command detector to detect malicious commands, and pass the detection results to the large model;

[0069] Step S4: The big model generates the actual big model output expected by the user or malicious user warning information.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] The present invention utilizes the instruction records retained in the user access large model application system, expands the malicious instruction samples and normal samples trained by machine learning, uses the open source general large model, further improves the accuracy of the generated samples, and greatly improves the accuracy of malicious instruction detection through the deep neural malicious instruction detection model based on fusion features. Compared with the existing solutions, it has the following beneficial effects and advantages:

[0072] 1. Adopting a rule-based malicious instruction generation method, the scale effect of user instruction records is fully utilized to solve the problem that the existing deep learning-based model lacks large-scale labeled actual instruction samples;

[0073] 2. Use multiple general large models to screen the categories of generated instruction samples again, which improves the quality of the constructed instruction dataset;

[0074] 3. Adopt a deep neural malicious command detection model based on fusion features, which integrates the multi-dimensional features of user commands to improve the effect of malicious command detection;

[0075] 4. The regular periodic model incremental update mechanism can realize the continuous learning and iteration of deep learning model parameters without interfering with normal business. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The schematic diagram of a self-supervised large-model malicious command detection system;

[0077] Figure 2 This is the workflow diagram of the malicious instruction detector;

[0078] Figure 3 It is the workflow diagram of the instruction sample generator;

[0079] Figure 4 It is the workflow diagram of the instruction filter;

[0080] Figure 5 This is a schematic diagram of the prompt word template. DETAILED DESCRIPTION

[0081] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0082] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0083] Embodiment 1

[0084] First of all, it should be noted that the purpose of big model malicious instruction detection is to prevent users from using big models to generate harmful content, such as false information, network attack codes, privacy violations or other illegal activities. By detecting and blocking malicious instructions, the security, legality and ethics of big models can be ensured, thereby safeguarding the interests of normal users and big model application providers.

[0085] The main problems to be solved in this embodiment are as follows:

[0086] 1. How to use the existing user instructions in the large model application system to generate real malicious instruction samples to solve the problem of lack of real training samples in machine learning;

[0087] 2. How to implement self-supervised learning in the malicious command detection model, solve the problem of updating and iterating the weights of the intelligent model, and improve the ability to detect malicious commands;

[0088] 3. How to integrate and aggregate the features of user commands to solve the multi-dimensional representation of user command features and improve the effectiveness of malicious command detection.

[0089] In view of the above problems, this embodiment proposes a self-supervised large-model malicious instruction detection system. Figure 1 ,include:

[0090] User terminal, firewall, instruction feature extractor, malicious instruction detector and target large model; user terminal, firewall, instruction feature extractor, malicious instruction detector and target large model are connected to the network through networking to form the large model malicious instruction detection system proposed in this embodiment;

[0091] in:

[0092] The user terminal asks questions to the target large model by requesting the large model application service port, and filters out invalid or unauthorized access through firewall rules;

[0093] The instruction feature extractor is used to extract instruction features of the input text of the question;

[0094] The malicious instruction detector performs malicious instruction detection based on the instruction features extracted by the instruction feature extractor, and transmits the detection results to the target big model, which generates the actual big model output expected by the user or malicious user warning information.

[0095] In this embodiment, specifically, it also includes:

[0096] Instruction sample generator and instruction filter;

[0097] The instruction feature extractor can also extract instruction features from the user history access record library, and the instruction features extracted by the instruction feature extractor are stored in the instruction feature library;

[0098] The instruction sample generator can automatically generate specific malicious instructions or normal instructions with similar semantics but different forms based on the rule engine and instruction feature library;

[0099] The instruction filter is used to filter the instructions generated by the instruction sample generator, and add the filtering results to the instruction sample library for training and updating the malicious instruction detector.

[0100] In this embodiment, specifically, Figure 2 As shown, the malicious instruction detector is internally loaded with a deep neural malicious instruction detection model based on fusion features, and the deep neural malicious instruction detection model includes: a feature fusion network, a malicious instruction discrimination network, and a model update mechanism;

[0101] The feature fusion network can fuse instruction sample features of different dimensions;

[0102] The malicious instruction identification network can classify the fused features into multiple categories, identify whether it is a malicious instruction, and output the category of the malicious instruction;

[0103] The model update mechanism can realize incremental updates of the model without affecting business operations.

[0104] In this embodiment, specifically, Figure 3 As shown, the instruction sample generator internally maintains an instruction generation module based on a rule engine; two types of instruction generation rule templates are defined in the rule engine, one is malicious instruction rule sample generation, the purpose of which is to generate malicious instructions with specific categories through normal user instructions; the other is neutral rule sample generation, the purpose of which is to generate a batch of instructions with the same functions as the original instructions but different character forms without changing the original instruction semantics, so as to expand the number of instruction samples.

[0105] Since the instruction sample generator uses an automated method to construct training samples, it is inevitable that there will be problems such as incorrectly classified malicious instructions, or normal instruction samples containing obscure malicious instructions. The function of the instruction filter is to filter the instructions generated by the instruction sample generator to ensure the accuracy of the instruction label, and then indirectly improve the accuracy of the malicious instruction detection method.

[0106] In this embodiment, specifically, the instruction filter has an open source universal big model, and the instruction filtering is performed through the open source universal big model.

[0107] In this embodiment, specifically, the extraction method of the instruction feature extractor is as follows:

[0108] First, the instructions are segmented to obtain instruction fragments. It should be noted that different segmentation methods are used according to the type of instructions. For example, for the conversation record, it can be segmented according to the timestamp of each user input and system response to obtain paired instruction fragments.

[0109] Then preprocess the instruction fragment; the preprocessing includes: removing noise, removing illegal characters, removing duplication, and standardizing; for example, for access records containing comments, regular expressions can be used to remove all non-Unicode encoded characters;

[0110] Finally, the string features and word features are extracted for the preprocessed instruction fragments, and the instruction feature extraction is completed using a natural language processing tool; preferably, the natural language processing tool is word2vec.

[0111] In this embodiment, please refer to Figure 3 , the generation process of the instruction sample generator is as follows:

[0112] Get instruction snippets from the instruction signature library and the corresponding eigenvectors;

[0113] Randomly select instruction generation rules according to the category requirements of instruction sample generation , generate rules according to instructions Preconditions for execution, select instruction fragments The target of a certain behavior rule execution If there is no matching target, the instruction generation rule is changed. If all instruction rules are executed and no target is found, the next instruction fragment is changed. Continue execution;

[0114] On target , execute instruction generation rules , generate malicious instruction samples of corresponding categories Or normal instruction sample ; thereby generating a malicious instruction sample set or a normal instruction sample set;

[0115] Label the malicious instruction sample set and the normal instruction sample set according to the sample category and add them to the instruction security detection sample library;

[0116] It should be noted that the generation categories of malicious command samples refer to the classification and definition of malicious user commands in the national standard "Basic Requirements for Generative Artificial Intelligence Service Security", the international standards "Generative Artificial Intelligence Application Security Testing Standard" and "Large Language Model Security Testing Method";

[0117] The following uses the generation of malicious instruction samples of the privacy stealing category to illustrate the principle of generating malicious instruction samples:

[0118] 1. In the historical instruction pairs of the user, retrieve the user's personal information keywords such as "account", "phone", "ID", "card number", "password", etc., and select the specific character positions according to the string regular matching method as the target for the sample generation rule execution;

[0119] 2. In the construction of the account information privacy theft sample, replace the original characters at the above target position with "obtain user account information" or "obtain user password" or "steal account credentials", and add it to the privacy theft malicious instruction sample collection;

[0120] 3. In the construction of financial information privacy theft samples, replace the original characters at the above target positions with "obtain user bank account" or "obtain credit card details" or "download payment records" and add them to the privacy theft malicious instruction sample collection;

[0121] 4. In the construction of the privacy-stealing sample of communication records, replace the original characters at the above target positions with "get call records", "download SMS content", "steal emails", and add them to the privacy-stealing malicious instruction sample collection.

[0122] Some rules and principles for generating normal instruction samples are as follows:

[0123] 1. Build a synonym list, adjective list, and modal word list of user commands. These word lists are used to amplify normal command samples;

[0124] 2. Synonym replacement: replace some words in a user command with synonyms. For example, change search to query; the command: help me search for a Beijing travel guide. Rewrite it into the command: help me search for a Beijing travel guide;

[0125] 3. Add adjectives. Add adjectives before the verbs in a user command. For example, the command: Help me summarize the history of the United States; change it to: Help me briefly summarize the history of the United States;

[0126] 4. Modal particle enhancement: add a modal particle at the end of a user command, for example: command: help me find the top ten scenic spots in China; rewrite it into command: help me find the top ten scenic spots in China;

[0127] 5. Sentence back translation: call a translation engine to translate the user's instructions into English, and then call another translation engine to translate the user's instructions into Chinese.

[0128] In this embodiment, please refer to Figure 4 , the screening process of the instruction filter is as follows:

[0129] Obtain the constructed malicious instruction samples from the instruction security detection sample library , Normal instruction sample ;

[0130] The malicious command sample , Normal instruction sample Through the prompt word template, input it into the open source general large model (for example: qwen2, chatglm4, llama3, etc.), and collect the detection results of the open source general large model; it should be noted that the prompt word template is Figure 5 As shown;

[0131] For input is a normal instruction sample , assuming that the detection results of the open source general large model mostly do not detect maliciousness, then the normal instruction samples generated For correctly labeled samples, add them to the instruction sample library; otherwise, discard them;

[0132] The input is a malicious instruction sample ; Assuming that the detection results of the open source general large model mostly detect specific malicious categories, the generated malicious instruction samples For correctly labeled samples, add them to the instruction sample library; otherwise, discard them.

[0133] In this embodiment, please refer to Figure 2 The feature fusion network uses three types of features: character level, word level, and sentence level, and uses the embedding method to convert the text into a vector form; define Represents three feature fusion networks; It is each layer of the feature fusion network; is the input vector of the feature fusion network, including: character-level embedding feature vector, word-level embedding feature vector and sentence embedding feature vector; Represents input to The vector of layers; is a hyperparameter; yes The output of the layer; express The weight of the layer; yes Layer offset; neural network activation function Using a variant of the linear rectification function Leaky ReLU, the forward transfer formula of the feature fusion network is:

[0134]

[0135]

[0136]

[0137] in:

[0138] Represents the 0th layer output of the feature fusion network;

[0139] Represents the activation function The output of , that is, when it is greater than zero, output the value of the original function, otherwise, output 0;

[0140] definition It is a malicious instruction to identify each layer of the network. Represents input to The vector of layers, yes The output of the layer, express The weight of the layer, yes Layer offset, neural network activation function The variant linear rectification function Leaky ReLU is also used. is the output function of the malicious instruction discrimination network. Since it is a multi-classification problem, the softmax function is used; the forward transfer formula of the malicious instruction discrimination network is:

[0141]

[0142]

[0143]

[0144]

[0145] in:

[0146] Represents the input vector of the multi-layer neural network;

[0147] Represents the linear combination result of the first layer, Similarly;

[0148] represents the output of the 4th layer, with the softmax function applied;

[0149] represents the first The index of the neuron;

[0150] represents a natural constant;

[0151] Indicates the malicious command identification network The linear combination result of neurons;

[0152] Represents the index of a neuron in the output layer;

[0153] represents the total number of neurons in the output layer;

[0154] Indicates The linear combination result of neurons;

[0155] use represents the label of the training data, represents the output label of the malicious instruction identification network; then use Find the index of the maximum value of the parameter as the output of the malicious instruction detector; the cross entropy loss function used in the training process of the deep neural malicious instruction detection model based on fusion features is as follows:

[0156]

[0157] in:

[0158] Represents the cross entropy loss function, which is used to evaluate the match between the predicted output of the classification model and the true label;

[0159] Represents the index of the data sample, that is, represents each malicious instruction sample;

[0160] Represents the index of the maximum value, which is used to select the category with the highest predicted probability as the final output of the model.

[0161] In this embodiment, it should also be noted that in order to achieve continuous learning and iteration of deep learning model parameters, the solution adopts an offline model incremental update mechanism, and its working principle is as follows:

[0162] 1. Define the malicious instruction discrimination model for online work on the i-th day as , the offline malicious instruction identification model is ;

[0163] 2. Collect the instruction fragments with labeled information on the i-th day from the instruction sample library for incremental update training of the offline model ;

[0164] 3. Use the offline model at the time when the business volume is low on the i-th day (for example: 3 am) Replace the online model, i.e. .

[0165] Based on the above-mentioned self-supervised large-model malicious instruction detection system, this embodiment also proposes a self-supervised large-model malicious instruction detection method, which specifically includes the following steps:

[0166] Step S1: The user terminal asks questions to the target large model by requesting the large model application service port, and filters out invalid or unauthorized access through firewall rules;

[0167] Step S2: using an instruction feature extractor to extract instruction features of the input text of the question in real time;

[0168] Step S3: Input the command features extracted in step S2 into the malicious command detector to detect malicious commands, and pass the detection results to the large model;

[0169] Step S4: The big model generates the actual big model output expected by the user or malicious user warning information.

[0170] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0171] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A self-supervised large-model malicious instruction detection system, characterized in that: include: User terminal, firewall, instruction feature extractor, malicious instruction detector and target large model; The user terminal asks questions to the target large model by requesting the large model application service port, and filters out invalid or unauthorized access through firewall rules; The instruction feature extractor is used to extract instruction features of the input text of the question; The malicious instruction detector performs malicious instruction detection based on the instruction features extracted by the instruction feature extractor, and transmits the detection result to the target large model, and the target large model generates the actual user-expected large model output or malicious user warning information; Also includes: Instruction sample generator and instruction filter; The instruction feature extractor can also extract instruction features from the user history access record library, and the instruction features extracted by the instruction feature extractor are stored in the instruction feature library; The instruction sample generator can automatically generate specific malicious instructions or normal instructions with similar semantics but different forms based on the rule engine and instruction feature library; The instruction filter is used to filter the instructions generated by the instruction sample generator, and add the screening results to the instruction sample library for training and updating the malicious instruction detector; The malicious instruction detector is internally loaded with a deep neural malicious instruction detection model based on fusion features, and the deep neural malicious instruction detection model includes: a feature fusion network, a malicious instruction discrimination network, and a model update mechanism; The feature fusion network can fuse instruction sample features of different dimensions; The malicious instruction identification network can classify the fused features into multiple categories, identify whether it is a malicious instruction, and output the category of the malicious instruction; The model update mechanism can realize incremental updates of the model without affecting business operations; The instruction sample generator internally maintains an instruction generation module based on a rule engine; two types of instruction generation rule templates are defined in the rule engine, one type is malicious instruction rule sample generation, the purpose of which is to generate malicious instructions with specific categories through normal user instructions; the other type is neutral rule sample generation, the purpose of which is to generate a batch of instructions with the same functions as the original instructions but different character forms without changing the original instruction semantics, so as to expand the number of instruction samples; The instruction filter has an open source general large model, and performs instruction filtering through the open source general large model; The extraction method of the instruction feature extractor is as follows: First, split the instruction to obtain instruction fragments; Then preprocessing the instruction fragments; the preprocessing includes: removing noise, removing illegal characters, removing duplication, and standardizing; Finally, we extract string features and word features from the preprocessed instruction fragments, and use natural language processing tools to complete instruction feature extraction. The generation process of the instruction sample generator is as follows: Get instruction snippets from the instruction signature library and the corresponding eigenvectors; Randomly select instruction generation rules according to the category requirements of instruction sample generation , generate rules according to instructions Preconditions for execution, select instruction fragments The target of a certain behavior rule execution If there is no matching target, the instruction generation rule is changed. If all instruction rules are executed and no target is found, the next instruction fragment is changed. Continue execution; On target , execute instruction generation rules , generate malicious instruction samples of corresponding categories Or normal instruction sample ; thereby generating a malicious instruction sample set or a normal instruction sample set; Label the malicious instruction sample set and the normal instruction sample set according to the sample category and add them to the instruction security detection sample library; The screening process of the instruction filter is as follows: Obtain the constructed malicious instruction samples from the instruction security detection sample library , Normal instruction sample ; The malicious command sample , Normal instruction sample Input the prompt word template into the open source general large model to collect the detection results of the open source general large model; For input is a normal instruction sample , assuming that the detection results of the open source general large model mostly do not detect maliciousness, then the normal instruction samples generated For correctly labeled samples, add them to the instruction sample library; otherwise, discard them; The input is a malicious instruction sample ; Assuming that the detection results of the open source general large model mostly detect specific malicious categories, the generated malicious instruction samples For correctly labeled samples, add them to the instruction sample library; otherwise, discard them; The feature fusion network uses three types of features: character level, word level, and sentence level, and uses an embedding method to convert text into a vector form; define Represents three feature fusion networks; It is each layer of the feature fusion network; is the input vector of the feature fusion network, including: character-level embedding feature vector, word-level embedding feature vector and sentence embedding feature vector; Represents input to The vector of layers; is a hyperparameter; yes The output of the layer; express The weight of the layer; yes Layer offset; neural network activation function Using a variant of the linear rectification function Leaky ReLU, the forward transfer formula of the feature fusion network is: in: Represents the 0th layer output of the feature fusion network; Represents the activation function The output of , that is, when it is greater than zero, output the value of the original function, otherwise, output 0; definition It is a malicious instruction to identify each layer of the network. Represents input to The vector of layers, yes The output of the layer, express The weight of the layer, yes Layer offset, neural network activation function The variant linear rectification function Leaky ReLU is also used. is the output function of the malicious instruction discrimination network. Since it is a multi-classification problem, the softmax function is used; the forward transfer formula of the malicious instruction discrimination network is: in: Represents the input vector of the multi-layer neural network; Represents the linear combination result of the first layer, Similarly; represents the output of the 4th layer, with the softmax function applied; represents the first The index of the neuron; represents a natural constant; Indicates the malicious command identification network The linear combination result of neurons; Represents the index of a neuron in the output layer; represents the total number of neurons in the output layer; Indicates The linear combination result of neurons; use represents the label of the training data, represents the output label of the malicious instruction identification network; then use Find the index of the maximum value of the parameter as the output of the malicious instruction detector; the cross entropy loss function used in the training process of the deep neural malicious instruction detection model based on fusion features is as follows: in: Represents the cross entropy loss function, which is used to evaluate the match between the predicted output of the classification model and the true label; Represents the index of the data sample, that is, represents each malicious instruction sample; Represents the index of the maximum value, which is used to select the category with the highest predicted probability as the final output of the model.

2. A self-supervised large-model malicious instruction detection method, characterized in that: A self-supervised large-model malicious instruction detection system based on claim 1, comprising: Step S1: The user terminal asks questions to the target large model by requesting the large model application service port, and filters out invalid or unauthorized access through firewall rules; Step S2: using an instruction feature extractor to extract instruction features of the input text of the question in real time; Step S3: Input the command features extracted in step S2 into the malicious command detector to detect malicious commands, and pass the detection results to the large model; Step S4: The big model generates the actual big model output expected by the user or malicious user warning information.