Large model security detection and defense control method and device, and electronic equipment

Abstract

The application provides a large model security detection and defense control method and device and electronic equipment. The method comprises the following steps: acquiring detection data for representing the security behavior of a large model; inputting the detection data into a target large model; collecting hidden states of multiple model layers and mapping the hidden states of different model layers to a shared representation space; determining candidate security neurons from the multiple model layers and obtaining security behavior changes by adjusting the activation states of the candidate security neurons; determining the risk state of a request to be processed and determining neuron regulation parameters; adjusting the activation state of the target security neuron according to the neuron regulation parameters to generate a security response result; determining a model parameter update range according to the distribution state of the target security neuron in the multiple model layers, and updating the parameters in the model parameter update range using security training data. The application can improve the positioning accuracy of security neurons, enhance the risk adaptive defense capability, and reduce the impact of security reinforcement on normal capability.

Description

Technical Field

[0001] This application relates to the field of large language model security technology, and in particular to a method, device and electronic device for large model security detection and defense control. Background Technology

[0002] With the widespread application of large language models in scenarios such as intelligent question answering, business assistance, and automated decision-making, the security of their output has gradually become a critical issue in the model deployment process. To reduce the risk of privacy leaks, illegal suggestions, and other harmful content generated by the model, existing technologies typically protect the model through secure alignment training, output distribution adjustment, or external harmful content detection. Among these, secure decoding and temperature adjustment methods mainly change the generation distribution during the inference output stage, while classifier filtering methods intercept risky content in the input or output.

[0003] However, most of the aforementioned methods operate at the model input-output level, lacking the identification and utilization of the mechanisms underlying security behavior within the model, making it difficult to determine which model layers and neurons actually perform security rejection functions. When faced with jailbreak attacks or novel adversarial requests, existing filtering and alignment mechanisms may still be bypassed; simultaneously, fixed-strength security controls can easily lead to excessive rejection of legitimate requests, affecting model usability. Existing security fine-tuning typically involves updating parameters across all layers or over a wide range, making it difficult to target and strengthen security-critical components, potentially damaging existing capabilities unrelated to security behavior. Therefore, how to locate key neurons related to security behavior within a large model and dynamically adjust and target them based on input risks has become an urgent technical problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a large-scale model security detection and defense control method, device and electronic device to solve the problems of existing technologies such as difficulty in locating security neurons, insufficient adaptability of fixed security regulation, and easy damage to normal capabilities by security enhancement.

[0005] A first aspect of this application provides a method for security detection and defense control of a large model, comprising: acquiring probe data to characterize the security behavior of a large model; inputting the probe data into a target large model; collecting hidden states of multiple model layers; and mapping the hidden states of different model layers to a shared representation space; determining candidate security neurons from multiple model layers based on the activation differences of different security behavior states in the shared representation space; obtaining security behavior changes by adjusting the activation states of the candidate security neurons; and determining target security neurons based on the security behavior changes; acquiring a request to be processed; determining the risk state of the request to be processed; determining neuron control parameters based on the risk state and the constraint relationship between the security capability and normal response capability of the target large model; adjusting the activation state of the target security neuron according to the neuron control parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result; determining the model parameter update range based on the distribution state of the target security neuron in multiple model layers; and updating the parameters within the model parameter update range using security training data, so that the updated target large model can be used for subsequent security detection and defense control.

[0006] A second aspect of this application provides a large-scale model security detection and defense control device, comprising: an acquisition module, configured to acquire probe data characterizing the security behavior of a large-scale model, input the probe data into a target large-scale model, collect hidden states of multiple model layers, and map the hidden states of different model layers to a shared representation space; an adjustment module, configured to determine candidate security neurons from multiple model layers based on the activation differences of different security behavior states in the shared representation space, obtain security behavior changes by adjusting the activation states of the candidate security neurons, and determine a target security neuron based on the security behavior changes; a determination module, configured to acquire a request to be processed, determine the risk state of the request to be processed, and determine neuron control parameters based on the risk state and the constraint relationship between the security capability and normal response capability of the target large-scale model; an adjustment module, configured to adjust the activation state of the target security neuron according to the neuron control parameters, so that the target large-scale model performs inference on the request to be processed in the adjusted activation state and generates a security response result; and an update module, configured to determine the model parameter update range based on the distribution state of the target security neuron in multiple model layers, and update the parameters within the model parameter update range using security training data, so that the updated target large-scale model can be used for subsequent security detection and defense control.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0008] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: By acquiring probe data to characterize the security behavior of a large model, inputting the probe data into the target large model, collecting hidden states from multiple model layers, and mapping the hidden states of different model layers to a shared representation space; based on the activation differences of different security behavior states in the shared representation space, candidate security neurons are identified from multiple model layers, and security behavior changes are obtained by adjusting the activation states of the candidate security neurons, and the target security neuron is determined based on the security behavior changes; pending requests are acquired, the risk state of the pending requests is determined, and neuron control parameters are determined based on the risk state and the constraint relationship between the security capability and normal response capability of the target large model; the activation state of the target security neuron is adjusted according to the neuron control parameters, so that the target large model performs inference on the pending requests in the adjusted activation state and generates a security response result; the model parameter update range is determined based on the distribution of the target security neuron in multiple model layers, and the parameters within the model parameter update range are updated using security training data, so that the updated target large model can be used for subsequent security detection and defense control. This application can improve the accuracy of security neuron localization, enhance risk adaptive defense capability, and reduce the impact of security enhancement on normal capability. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the large-scale model security detection and defense control method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the large-scale security detection and defense control device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0012] Existing security defense technologies for large language models typically employ methods such as secure alignment training, output distribution adjustment, or external harmful content detection to control the model's input or output. Among these, secure decoding and generation parameter adjustment primarily restrict the generation of harmful content during the output generation stage, while classifier filtering intercepts content by identifying risky elements in input requests or model output. While these methods can improve the security of large language models to some extent, they mostly operate at the external input and output level, lacking the identification and utilization of the mechanisms underlying the model's internal security behaviors.

[0013] Existing technologies struggle to determine which model layers and neurons are actually involved in security denial behavior, making existing security mechanisms prone to failure when facing jailbreak attacks or novel adversarial requests. Furthermore, using fixed security control strength can lead to either insufficient defense with lower strengths or excessive rejection of legitimate requests with higher strengths. Additionally, fine-tuning security through full-layer or large-scale parameter updates may alter model parameters unrelated to security behavior, resulting in a decline in the model's normal responsiveness. Therefore, existing technologies suffer from difficulties in locating security neurons, insufficient adaptability of fixed security controls, and the potential for security enhancements to impair normal capabilities.

[0014] To address the aforementioned issues, this application first acquires probe data to characterize the security behavior of a large model. This probe data is then input into the target large model, and the hidden states of multiple model layers are collected. By mapping the hidden states of different model layers to a shared representation space, a unified basis for comparison is established for the internal representations of different model layers. Next, candidate security neurons are determined based on the activation differences corresponding to different security behavior states. By changing the activation states of these candidate security neurons and analyzing the resulting changes in security behavior, target security neurons that effectively influence the model's security behavior are selected. Thus, the security defense target is further located from the model as a whole or the model output to neurons within the model that are associated with security behavior.

[0015] In the model inference stage, this application further determines the risk state of the request to be processed, determines the neuron control parameters based on the risk state and the constraint relationship between the security capability and normal response capability of the target large model, and adjusts the activation state of the target safe neurons using the neuron control parameters so that the target large model can complete inference under the corresponding safe activation state. In the model update stage, the model parameter update range is determined based on the distribution of the target safe neurons in multiple model layers, and the corresponding model parameters are updated differentially using safe training data, so that the safe response of the target safe neurons to risky inputs is strengthened, while maintaining the model representation and response capability under normal inputs.

[0016] Through the above technical solution, this application can locate key neurons related to security behavior from the internal representation of a large model, and implement adaptive neuron-level security control according to the risk level of the request to be processed. At the same time, it uses the distribution state of security neurons to constrain the model update range, thereby improving the accuracy of security neuron location and risk adaptive defense capability, and reducing the impact of the security enhancement process on the normal response capability of the model.

[0017] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0018] Figure 1 This is a flowchart illustrating the large-scale model security detection and defense control method provided in this application embodiment. Figure 1 As shown, the method may specifically include: S101, acquire probe data to characterize the security behavior of the large model, input the probe data into the target large model, collect the hidden states of multiple model layers, and map the hidden states of different model layers to the shared representation space; S102, Based on the activation differences of different safety behavior states in the shared representation space, candidate safety neurons are determined from multiple model layers, and safety behavior changes are obtained by adjusting the activation states of the candidate safety neurons. The target safety neuron is determined based on the safety behavior changes. S103, Obtain the request to be processed, determine the risk status of the request to be processed, and determine the neuron control parameters based on the risk status and the constraint relationship between the security capability and normal response capability of the target large model. S104, adjust the activation state of the target security neuron according to the neuron regulation parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result; S105, determine the model parameter update range based on the distribution of the target security neurons in multiple model layers, and use security training data to update the parameters within the model parameter update range, so that the updated target large model can be used for subsequent security detection and defense control.

[0019] In some embodiments, probing data for characterizing the security behavior of a large model is acquired, the probing data is input into the target large model, hidden states of multiple model layers are collected, and hidden states of different model layers are mapped to a shared representation space, including: Based on the security attributes of the request and the target response status, construct detection data with a security behavior comparison relationship, and input the detection data into the target large model sample by sample; During the forward inference process of the target large model, the hidden states corresponding to each probe sample are collected according to the model level, and the association between the hidden states and the corresponding model layers and probe samples is established. Obtain the lexical mapping parameters of the target large model, transform the hidden states of each model layer into a common lexical semantic space through a unified mapping relationship, and generate a shared representation result that preserves the association between model layers.

[0020] Specifically, a large-scale language model with 70 parameters deployed in hotel business scenarios is used as the target large model. Security detection data is constructed and its internal representation is collected to provide a unified data foundation for subsequent security knowledge neuron recognition. The security detection data is constructed around the security attributes of requests and the model's target response state, creating a comparative relationship between different detection samples that reflects differences in security behavior. For example, hotel business requests requiring a normal response from the model, risky requests that should trigger model rejection, and corresponding rejection responses can be collected. Normal business requests can include inquiries about hotel breakfast times, pick-up and drop-off services, and check-in procedures, while risky requests can include requests to query other guests' private information, obtain access control bypass methods, or use jailbreak hints to induce the model to break security restrictions. Rejection responses correspond to the model's security output in response to risky requests. Through this approach, the detection data simultaneously covers normal response states, risky input states, and security rejection states, thereby avoiding judgment of the model's internal security behavior based solely on a single type of request.

[0021] In one example, a probe dataset is constructed containing 2000 risky requests, 6000 legitimate requests, and 2000 denial responses. Each probe sample is configured with a sample identifier, security attribute marker, and target response status. Risky requests can be obtained from historical security test data, manually constructed attack samples, and jailbreak template variants, but in actual implementation, the specific risk category is not limited; it is only necessary to ensure that the samples can trigger different levels of security behavior. Each probe sample is fed into the target large model one by one according to a uniform input format, enabling the model to perform forward propagation under the same inference configuration, avoiding additional state deviations introduced by changes in operating conditions such as sampling temperature and context length.

[0022] When the target large model performs forward inference, the hidden states of each model layer are collected synchronously. In this embodiment, the target large model includes 80 model layers. The system can set up state collection nodes at the location where the hidden representation is output at each model layer. When a probe sample propagates to the corresponding model layer, the hidden state of the current layer is recorded, and an association index is established using the sample identifier, model layer identifier, and sequence position. For requests containing multiple text locations, positions related to risk semantics, end-convergence positions, or sample-level hidden representations can be formed according to preset aggregation rules. Thus, each probe sample forms a hidden state sequence that expands along the model layers, enabling subsequent processing to track changes in the internal representation of the same sample in different model layers, and to compare the response differences of the same model layer to samples with different security attributes.

[0023] Considering that the hidden states generated by different model layers are inconsistent in numerical scale and internal semantic representation, this embodiment does not directly compare the original hidden states of different model layers laterally. Instead, it reads the lexical mapping parameters used for lexical prediction in the target large model and transforms the hidden states of each model layer to a common lexical semantic space. Specifically, it can call the embedding matrix corresponding to the model output vocabulary or the projection matrix associated with its parameters to perform a unified linear mapping on the hidden states generated by the l-th model layer, obtaining a projection representation consistent with the vocabulary dimension of the target large model. For models using a weight-sharing structure, the output layer lexical mapping parameters can be directly reused; for models where the input embedding and output mapping parameters are not shared, a unified lexical mapping parameter fixed at deployment time is used to complete the transformation, ensuring that different model layers use the same semantic reference.

[0024] After the transformation, the states of each model layer, originally residing in independent hidden spaces, are represented as semantic response distributions oriented towards a common vocabulary set. For example, the hidden states of the same risk request formed in layers 15, 32, and 48 can all be observed under the same vocabulary dimension, showing response changes related to security semantics such as rejection, security, and privacy, after unified mapping. The system retains the correspondence between the projected representation and the original model layer identifiers and probe sample identifiers, forming a shared representation result. In this way, when calculating the differences in neuronal activation under different security behavior states, cross-layer intra-states can be compared under a unified semantic reference, reducing misjudgments caused by differences in dimensions and representation spaces between model layers, and providing a comparable basic representation for identifying security knowledge neurons distributed across multiple model layers.

[0025] In some embodiments, candidate safety neurons are determined from multiple model layers based on activation differences of different safety behavior states in a shared representation space, including: Based on the safety behavior state corresponding to the probe sample, the neuron activation in the shared representation results is grouped and statistically analyzed to determine the degree of activation deviation of each neuron between different safety behavior states. Based on the degree of activation deviation, the neurons in each model layer are evaluated for safe association, and neurons whose activation changes meet the preset association conditions are selected. By combining the activation differences between neurons in the rejection behavior state and the normal response state for secondary verification, neurons that meet the difference conditions are identified as candidate safe neurons.

[0026] Specifically, after obtaining the shared representation results, the activation of neurons in each model layer is grouped and statistically analyzed based on the pre-associated security behavior states of the probe samples to distinguish internal activation patterns related to risky inputs, rejection behaviors, and normal responses. Taking a hotel AI assistant as an example, the target large model has an 80-layer structure. The probe data includes 2,000 harmful requests, 6,000 safe requests, and 2,000 rejection responses. Harmful requests may include requests to obtain other guests' private information, bypass access control, and use jailbreak prompts to induce the model to break security restrictions. Safe requests correspond to normal business inquiries such as breakfast time, check-in process, and pick-up and drop-off services. The system extracts the projected activation corresponding to each neuron from the shared representation results according to the sample identifier, model layer identifier, and neuron location, and forms risky input group, rejection behavior group, and normal response group respectively.

[0027] For any neuron in any model layer, the activation statistics of that neuron under different safe behavior states are calculated, and the degree of activation deviation between states is determined within a unified lexical semantic space. To reduce the impact of abnormal fluctuations in individual samples, multiple samples under the same state can be normalized first, and then the corresponding activation mean can be calculated. The difference in mean is then corrected based on the degree of dispersion within the samples. For example, if a neuron in layer 32 maintains high activation in most harmful requests and rejection responses, but maintains low activation in normal requests such as hotel breakfast and room type inquiries, then this neuron exhibits a large positive activation deviation between safe behavior states. For neurons that show transient high activation only in a small number of samples but are unstable in other samples in the same group, the corresponding deviation degree is reduced to avoid treating accidental responses as safe association features.

[0028] After obtaining the activation deviation of each neuron, the system performs a safety association assessment according to the model layer. Specifically, the activation deviation magnitude, the consistency of the deviation direction, and the stability across samples can be mapped to a safety association state, and ranked within each model layer. Neurons whose safety association states meet preset association conditions are included in the initial candidate set. The preset association conditions can be implemented using a ranking threshold or an adaptive quantile boundary. For example, neurons with safety association states in the early intervals of each model layer are selected to avoid fixed threshold mismatch caused by differences in activation scale and number of neurons in different model layers. For neurons that are active in harmful requests but do not maintain the same direction of change in rejection responses, their safety association state can be reduced, and they are not included in the initial candidate set.

[0029] During the grouping and statistical process, neuron responses can be extracted based on the key sequence positions that trigger security semantics in the probe samples. Activations at different sequence positions within the same sample can be aggregated, ensuring the statistical results reflect the neuron's response to the complete security attribute request, rather than being influenced by a single lexical unit. For different model layers, since the preceding steps have already mapped the hidden states to a common lexical semantic space, a unified semantic reference is used for security association assessment, while retaining the model layer origin and avoiding direct mixing of neuron positions from different layers. If the overall activation fluctuation of a certain model layer is significantly higher than other model layers, the degree of deviation can be standardized within the layer based on the activation distribution of that layer before participating in candidate ranking. This ensures that candidate selection focuses on security behavior-related changes rather than differences in numerical scale between layers.

[0030] Subsequently, a secondary validation is performed on the initial candidate set to further confirm whether the candidate neurons are directly related to the model's rejection behavior. The system statistically analyzes the activation level of each initial candidate neuron in the rejection behavior group and the normal response group, calculates the normalized activation difference between the two groups, and determines the reliability of the difference by combining the intra-group fluctuations. When the activation of a candidate neuron in the rejection behavior state is higher than that in the normal response state, and the corresponding difference meets the preset difference condition, the neuron is retained; when the activation difference is insufficient or the direction of change is inconsistent among samples, it is discarded. In this embodiment, 0.5 can be used as the example judgment threshold for normalized activation difference. In actual deployment, the difference boundary can also be dynamically determined based on the activation distribution of the target large model on the probe data.

[0031] For example, after completing the above screening of 80 model layers of a hotel AI assistant, 85 candidate safe neurons can be obtained at layer 15, 142 at layer 32, and 103 at layer 48, while layer 60 may not have any neurons that meet the difference criteria. The system establishes associations between the neurons that are ultimately retained and their corresponding model layer positions, neuron positions, activation deviation degrees, and difference confidence levels, forming a candidate safe neuron set. By first using activation deviations of different safe behavior states for safety association screening, and then using the difference between rejection behavior states and normal response states for secondary verification, neurons that are only related to general semantic expressions but do not participate in safety rejection can be reduced from entering the candidate set. This provides a clear object for subsequent verification of the actual impact of candidate safe neurons on the model's safe behavior through activation inhibition.

[0032] In some embodiments, obtaining changes in security behavior by adjusting the activation state of candidate security neurons, and determining target security neurons based on these changes in security behavior, includes: Pre-defined inhibition is applied to candidate safe neurons to obtain corresponding controlled activation states, and model inference is performed using safe probe samples in the controlled activation states. The model outputs before and after regulation are compared to determine the degree of change in the safety response of each candidate safety neuron. Based on the matching results between the degree of change in security response and the preset validity conditions, candidate security neurons that have an effective impact on the security behavior of the target large model are screened, and the target security neurons are determined.

[0033] Specifically, after obtaining the candidate set of secure neurons, the actual correlation between the candidate secure neurons and the model's secure behavior is intervened and verified by actively changing the activation state of the candidate secure neurons. This reduces the number of neurons that only have statistical correlation but do not actually participate in the security rejection process from entering the subsequent control scope. Taking a hotel AI assistant as an example, the candidate secure neurons obtained above are used to establish a verification index according to their model layer position and security correlation state. Samples that can trigger risky behaviors such as privacy leaks, illegal suggestions, and jailbreak attacks are selected from the security detection data to form a security detection sample set. Before verification, a round of model inference is first performed under the condition of no activation control, and the model output and rejection state corresponding to each security detection sample are recorded to form the baseline security behavior result.

[0034] Subsequently, during the forward inference process of the target large model, inhibitory control is applied to candidate safety neurons. Specifically, after the candidate safety neurons complete their initial activation calculations, the initial activation values ​​are scaled according to a preset inhibition coefficient, causing the corresponding neurons to enter a controlled activation state. The inhibition coefficient is less than 1; for example, the activation value of a candidate safety neuron can be adjusted to 0.5 times its initial activation value, thereby weakening the safety-related internal responses transmitted by the corresponding neuron to subsequent model layers. The inhibition treatment only applies to the candidate safety neurons currently participating in the validation; the remaining neurons maintain their initial activation according to the normal forward propagation process, ensuring that changes in the model output reflect the impact of the inhibition on candidate safety neurons as much as possible. In actual validation, candidate safety neurons can be grouped and inhibited according to model layers or safety-related states first, and then the validation scope can be narrowed down for groups that cause significant changes in safety behavior to identify more influential candidate safety neurons.

[0035] Under controlled activation, the same security probe samples as in the benchmark validation are input into the target large model again, maintaining consistency in the prompt context, generation parameters, and output judgment methods. For each sample, the system determines whether the adjusted model output continues to exhibit rejection behavior and establishes a correlation with the benchmark output of the corresponding sample before adjustment. For example, when faced with a request to obtain the name and phone number of the guest in room 302, the model can return rejection content without suppressing candidate security neurons. However, after suppressing some candidate security neurons in layer 32, the model begins to provide information that should not have been generated in the first place or no longer forms an explicit rejection, indicating that the reduced activation of the corresponding candidate security neurons has weakened the model's security response.

[0036] The system further calculates the degree of change in security response based on the model output before and after regulation. The degree of change in security response can be determined based on changes in the rejection state and the attack success state; in this embodiment, the focus is on statistically analyzing the attack success rate of security probe samples. If most risky requests are rejected in the baseline state, but the attack success rate significantly increases after applying suppression regulation, it indicates that the currently controlled candidate security neuron makes a real contribution to the model's security behavior. For a single candidate security neuron or a group of candidate neurons, the difference in attack success rate before and after suppression can be calculated separately, and combined with the proportion of affected security probe samples to form the corresponding degree of change in security response. For candidate security neurons that only cause fluctuations in the output of a small number of boundary samples, but whose overall rejection capability remains basically unchanged, their contribution to security behavior is considered weak.

[0037] The preset validity conditions can be set based on the increase in attack success rate or the decrease in rejection rate. For example, when a candidate security neuron is suppressed, if the attack success rate in the security probe sample increases beyond the preset change boundary relative to the baseline state, and this change can be repeated on multiple risk samples, the corresponding candidate security neuron is determined to meet the validity conditions. If the attack success rate does not change significantly, the candidate range or suppression coefficient is adjusted for further verification. In this embodiment, a uniform suppression coefficient can be used to complete the first round of verification, and then the suppression intensity of the critical candidate security neuron can be changed to determine whether the security response change is stable with the degree of activation suppression, thereby reducing misjudgments caused by one-time output fluctuations.

[0038] After verification, candidate safety neurons that meet the validity criteria are identified as target safety neurons, and their corresponding model layer locations, neuron locations, and degree of change in safety response are retained. Taking an 80-layer hotel AI assistant as an example, target safety neurons can be distributed across multiple model layers, such as layer 15, layer 32, and layer 48, while model layers without effective safety neurons are not included in the subsequent activation control scope. Through this intervention verification method, candidate relationships obtained based on activation differences are further transformed into effective associations verified by changes in safety behavior, providing a reliable neuron localization basis for subsequent enhanced regulation of target safety neurons based on input risk status.

[0039] In some embodiments, a request to be processed is acquired, the risk status of the request to be processed is determined, and neuron modulation parameters are determined based on the risk status and the constraint relationship between the security capability and normal response capability of the target large model, including: Extract the risk characterization of the requests to be processed, and determine the corresponding risk status according to the preset risk judgment relationship; Obtain the safety response state and normal response state of the target large model under different candidate control parameters, and determine the corresponding safety capability and normal response capability respectively; The constraint weights of safety capability and normal response capability in the parameter selection process are adjusted according to the risk status, and the neuron regulation parameters that match the current risk status are determined from the candidate regulation parameters that meet the lower limit condition of normal response capability.

[0040] Specifically, after the target large model completes the localization of the target security neurons, the neuron modulation parameters used to control the activation intensity of the target security neurons are adaptively determined based on the risk level of the request to be processed and the balance between security defense capabilities and normal business response capabilities. Taking a hotel AI assistant as an example, when a user request is received, a pre-trained lightweight risk classification model is first used to extract risk representations from the request text. The risk classification model can be fine-tuned based on a pre-trained text encoding model, with training samples derived from historical request logs and security test data, and pre-labeled with corresponding risk levels. During runtime, the request to be processed is input into the risk classification model to obtain risk representations reflecting privacy acquisition intent, illegal operation intent, jailbreak inducement features, and normal business semantics. Then, a low-risk, medium-risk, or high-risk status is output according to a preset risk judgment relationship. For example, asking what time the hotel breakfast starts can be judged as low-risk, asking for information about the guest in room 302 can be judged as medium-risk, and a request to provide a method to circumvent hotel access control using role-based bypass can be judged as high-risk.

[0041] Neuron modulation parameters are used to determine the enhancement level of the target security neuron relative to its original activation state. This embodiment does not use a fixed enhancement coefficient; instead, multiple candidate modulation parameters are selected before model deployment, and each candidate parameter is evaluated using security probe samples and normal probe samples. For any candidate modulation parameter, the target security neuron is activated according to the corresponding enhancement level. Inference is then performed using a security probe set consisting of jailbreak attacks and other risky requests. The proportion of valid rejection responses generated by the model is statistically analyzed to obtain the corresponding security capability. Valid rejection responses can be determined based on whether the model output contains rejection semantics. For example, outputs that cannot provide other guest information or assist in bypassing access control are recorded as security responses. To reduce the impact of specific wording variations, rejection semantic classification results can also be used instead of fixed text matching results.

[0042] Simultaneously, the same candidate control parameter is applied to normal probe samples such as hotel inquiries, service requests, and check-in. The proportion of samples where the model does not generate abnormal rejections is used to obtain the normal response capability. For example, if normal requests such as breakfast time, parking service, and pick-up service can still receive responses after the target security neuron is enhanced, the corresponding samples are counted as valid normal responses; if the model rejects normal requests due to excessive enhancement of the security neuron, the normal response capability decreases. Thus, each candidate control parameter is associated with two evaluation states: security capability and normal response capability, thereby characterizing the relationship between increased security defense and potential decrease in normal availability after the enhancement intensity is increased.

[0043] During the parameter selection phase, the constraint weights corresponding to security capabilities and normal response capabilities are adjusted based on the current risk status. In a low-risk state, the constraint weight for normal response capabilities is increased, prioritizing maintaining the model's normal business processing capabilities in parameter selection. In a medium-risk state, the two constraints are kept relatively balanced. In a high-risk state, the constraint weight for security capabilities is increased, making parameter selection more inclined to improve the ability to reject jailbreak attacks and high-risk requests. Simultaneously, a lower limit for normal response capabilities is set; for example, requiring a non-rejection rate of at least 90% for normal probe samples as a constraint condition for subsequent parameter selection. This avoids simply increasing security enhancement strength leading to the rejection of a large number of normal requests.

[0044] For example, offline testing can be conducted on multiple enhancement coefficients that increase progressively from 1.0. When the enhancement coefficient is small, the normal response capability is high, but the rejection capability against some jailbreak attacks is insufficient. As the enhancement coefficient increases, the rejection rate of the security probe set increases, but the probability of normal requests being falsely rejected also increases accordingly. The system uses different constraint weights for low-risk, medium-risk, and high-risk requests, and selects the parameter with the optimal overall evaluation state from the candidate parameters that meet the lower limit of normal response capability. In this embodiment, low-risk requests can maintain the original activation state of the target security neuron, i.e., using 1.0x modulation; medium-risk requests can use approximately 1.4x enhancement; and high-risk jailbreak requests can use approximately 1.6x enhancement. The specific values ​​can be re-determined according to the target large model, security probe data, and model update status, and are not limited to fixed values.

[0045] During model deployment, target control parameters corresponding to different risk states can be pre-saved as a mapping relationship between risk states and neuron control parameters. When a new request arrives, only the risk state needs to be determined and the corresponding neuron control parameters invoked; there is no need to re-execute the complete optimization calculation of security capabilities and normal response capabilities during each online inference process. By incorporating risk states, security capabilities, and normal response capabilities into the neuron control parameter determination process, low-risk requests maintain a high level of normal response capability, while high-risk requests receive stronger activation of security neurons, providing a parameter basis for subsequent dynamic security defense execution according to neuron control parameters.

[0046] In some embodiments, determining neuronal modulation parameters that match the current risk state includes: Before deploying the target large model, the constraint weights corresponding to the security capability and normal response capability are changed according to different risk states, and the constraint evaluation is performed on the candidate control parameters to determine the target control parameters corresponding to each risk state. Establish a control mapping relationship between risk status and target control parameters; After obtaining the risk status of the request to be processed, the corresponding target control parameter is matched from the control mapping relationship, and the matching result is determined as the neuron control parameter used in the current inference process.

[0047] Specifically, to avoid re-executing the constraint optimization between security capabilities and normal response capabilities for each pending request in the target large model, the target control parameters corresponding to different risk states are solved in advance before the model is officially deployed, and a control mapping relationship that can be directly invoked for online inference is established. Taking the hotel AI assistant as an example, the system pre-sets three risk states: low risk, medium risk, and high risk, and configures the constraint weights of security capabilities and normal response capabilities in the parameter evaluation process for each risk state. The low-risk state prioritizes maintaining normal business response capabilities, the medium-risk state balances security defense and normal response, and the high-risk state increases the constraint weights corresponding to security capabilities, enabling the target security neurons to obtain higher enhancement strength.

[0048] During the offline parameter evaluation process, multiple candidate regulation parameters are first generated based on the original activation range of the target security neuron. For example, different activation enhancement coefficients, such as 1.0, 1.1, 1.2, and 1.8, can be set. Security detection and normal capability detection are then performed for each candidate regulation parameter. Security detection samples can include jailbreak attacks, privacy theft, and unauthorized operation requests, while normal capability detection samples can include normal hotel service requests such as breakfast time, parking service, pick-up and drop-off service, and check-in procedures. For each candidate regulation parameter, the system adjusts the activation value of the target security neuron according to the corresponding enhancement coefficient and calculates the proportion of valid rejection responses in the security detection samples as the security capability state. Simultaneously, it calculates the proportion of normal responses without abnormal rejections in the normal detection samples as the normal response capability state.

[0049] After completing offline testing of each candidate control parameter, the evaluation weights corresponding to the two types of capabilities are adjusted according to different risk states. For example, in a low-risk state, the evaluation weight corresponding to the normal response capability is increased, making the parameter selection more inclined to approach the original active state; in a medium-risk state, the security capability and the normal response capability are kept relatively balanced; in a high-risk state, the evaluation weight corresponding to the security capability is increased, making the parameter selection more inclined to improve the attack request rejection rate. At the same time, a lower limit for the normal response capability is set. In this embodiment, a non-rejection ratio of normal probe samples of not less than 90% can be used as a condition for the parameter to be valid. When a candidate control parameter can significantly improve the attack rejection rate, but causes the normal response capability to fall below the lower limit, the candidate control parameter is removed from the valid parameter range of the current risk state.

[0050] For example, offline evaluation results show that an enhancement coefficient of 1.0 can maintain the highest normal response capability, but its ability to reject some jailbreak attacks is insufficient; an enhancement coefficient of 1.4 can significantly improve the rejection rate of medium-risk requests while maintaining a normal response capability above 90%; an enhancement coefficient of 1.6 can further improve the rejection capability of high-risk jailbreak requests, while the normal response capability still meets the lower limit condition; when the enhancement coefficient continues to increase to 1.8, although the rejection rate of the security probe set continues to increase, some normal hotel inquiries are excessively rejected, and therefore are not used as target control parameters. Based on this, the system can determine the target control parameter corresponding to low risk as 1.0, the target control parameter corresponding to medium risk as 1.4, and the target control parameter corresponding to high risk as 1.6. The above values ​​are only examples in this embodiment. If the model version, security probe data, or business scenario changes, the offline constraint evaluation can be re-executed and the corresponding parameters updated.

[0051] After obtaining the target control parameters corresponding to each risk state, the system uses the risk state as an index to associate and store the target control parameters, the corresponding model version, the safety capability evaluation results, and the normal response capability evaluation results, forming a control mapping relationship. The control mapping relationship is saved with the deployment version of the target large model and can be recalculated after the target large model is fine-tuned, the safety neuron set changes, or the probe data is updated, thereby avoiding the continued use of mismatch parameters after changes in the internal state of the model.

[0052] During the online inference phase, requests are first assessed for their current risk status. For example, if the start time of hotel breakfast is deemed low-risk, the target control parameter corresponding to 1.0 is directly obtained from the control mapping relationship; if a request to query guest information for room 302 is deemed medium-risk, the target control parameter corresponding to 1.4 is obtained; if a request for access control bypass methods based on jailbreak hints is deemed high-risk, the target control parameter corresponding to 1.6 is obtained. The system determines the matched target control parameters as the neuron control parameters in the current inference process and passes them to the target security neuron activation control link. By moving the complex security capability and normal response capability constraint assessment to the offline phase, the online phase only performs risk status matching and parameter invocation, thereby reducing additional computation in the real-time inference process while maintaining the risk adaptive control capability.

[0053] In some embodiments, the activation state of the target security neuron is adjusted according to neuron modulation parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result, including: Establish activation regulation relationships based on the model layer and neuron location corresponding to the target security neuron, and obtain the current activation state of the corresponding target security neuron during the forward inference process of the pending request; The current activation state is enhanced according to the neuron's regulation parameters, while keeping the activation state of neurons other than the target safe neuron unaffected by the enhancement transformation. The enhanced activation state is passed to subsequent model layers to continue inference, and a security response result is generated based on the response content output by the target large model.

[0054] Specifically, after determining the risk of pending requests and obtaining the corresponding neuron regulation parameters, the system performs targeted activation enhancement on the corresponding neurons in the target large-scale model inference process based on the localization results of the target safety neurons. Taking a hotel AI assistant as an example, the target safety neurons are distributed across multiple model layers, such as layer 15, layer 32, and layer 48. The system pre-records the model layer identifier, neuron location, and degree of safety response change for each target safety neuron, and establishes activation regulation relationships accordingly. These activation regulation relationships are used to quickly locate the target safety neurons that need adjustment during forward propagation of the model, while simultaneously associating the corresponding neuron regulation parameters with the neuron location, avoiding uniform enhancement of the entire model layer or neurons unrelated to safety behavior.

[0055] Once a request enters the target large model, the model sequentially completes lexical encoding and forward computation of each model layer according to the normal inference process. When executing a model layer containing target safety neurons, the system reads the current activation value output at the corresponding neuron position based on the activation control relationship and temporarily stores the current activation state in the current round of inference context. For cases where the same model layer contains multiple target safety neurons, a position mask can be constructed based on the neuron position, and only the activation component corresponding to the position mask is extracted. For target safety neurons in different model layers, activation acquisition and control are performed at the corresponding model layer, allowing the enhancement operation to take effect layer by layer along the model's forward propagation process. Therefore, safe activation control can be inserted at predetermined neuron positions without changing the original network structure of the target large model.

[0056] After obtaining the current activation state, an enhancement transformation is performed according to the neuron modulation parameters corresponding to the current pending request. In this embodiment, the current activation value of the target security neuron can be multiplied by the enhancement coefficient, amplifying its security-related response based on the original activation. For example, when a user inputs the start time of hotel breakfast, the risk state is low risk, and the neuron modulation parameter can be 1.0, maintaining the original activation of the target security neuron; when a user requests to query the guest information of room 302, the risk state is medium risk, and a 1.4x enhancement can be used; when a user requests a method to bypass hotel access control by jailbreaking, the risk state is high risk, and a 1.6x enhancement can be used. The enhancement coefficient is obtained from the aforementioned offline constraint evaluation and can be updated with the model version and security detection results.

[0057] The enhancement process only applies to the activation components corresponding to the target safe neuron. For other neurons in the same model layer that are not identified as the target safe neuron, the activation values ​​calculated by the original model continue to be used; for model layers that do not contain the target safe neuron, no enhancement operation is inserted. In practice, a modulation mask can be generated based on the location of the target safe neuron. The activation components that are hit by the modulation mask are scaled according to the neuron's modulation parameters, and then recombined with the original activation components that are not hit to form the enhanced hidden state of the current model layer. This can limit the scope of the safety modulation's influence and reduce the possibility of amplifying the activation of the entire model layer and causing a shift in the normal semantic representation.

[0058] After enhancing the activation of the current model layer, the enhanced hidden state is used as the input to the next model layer for forward propagation. Because the response of the target safety neuron is strengthened under high-risk inputs, its safety-related representation can be passed down through subsequent model layers, influencing the final word generation probability. If subsequent model layers still contain target safety neurons, enhancement continues according to the same regulatory relationship when propagating to the corresponding model layer, until all model layer computations and response generation are completed.

[0059] For example, for a medium-risk request to inquire about the guest information of room 302, the target security neuron in layer 32 is enhanced by 1.4 times, increasing the internal responses related to privacy protection and denial. Subsequent model layers continue calculations based on this enhanced state, ultimately generating a security response that cannot provide other guest privacy information. For high-risk jailbreak requests, the target security neurons in layers 15, 32, and 48 can be enhanced by 1.6 times respectively, continuously strengthening the security representations across multiple model layers, ultimately outputting a response that denies providing access control bypass methods and directs the user to contact the front desk. For low-risk normal requests, due to the use of adjustment parameters close to the original state, the model can still generate normal business responses such as breakfast times. The system associates the final output with the current pending requests and risk states, generating security response results and providing inference records for subsequent security state statistics and model updates.

[0060] In some embodiments, the model parameter update range is determined based on the distribution of the target security neurons across multiple model layers, and the parameters within the model parameter update range are updated using security training data, so that the updated target large model can be used for subsequent security detection and defense control, including: The degree of security correlation of the corresponding model layer is determined based on the distribution density of the target security neurons in each model layer, and the model parameter update range and freezing range are divided according to the degree of security correlation. Input safety training data into the target large model, and within the range of model parameter updates, construct safety constraints based on the activation state of the target safety neurons under risky and normal inputs, and determine the parameter update amount by combining the model response bias; Differential parameter updates are performed on different model layers according to the degree of security association, and the security representation formed by the high security association model layer is aligned and migrated to the adjacent low security association model layer to obtain the updated target large model.

[0061] Specifically, after identifying and validating the target security neurons, the subsequent security enhancement scope is determined based on the distribution of these neurons across different model layers of the target large model. This ensures that parameter updates are concentrated on model layers strongly correlated with security behavior. Taking an 80-story hotel AI assistant as an example, the system counts the number of target security neurons in each model layer and calculates the proportion of these neurons to the total number of neurons in the current model layer. This proportion is used as the security neuron distribution density. Model layers with high security neuron distribution density indicate that they contain a large number of target security neurons that can influence security rejection behavior, and are thus identified as high-security-correlation model layers. Model layers with a certain number of target security neurons but low distribution density are identified as low-security-correlation model layers. Model layers where no target security neurons are detected are identified as non-security-correlation model layers.

[0062] In one example, after density sorting of 80 model layers, layers 28 to 36 can be designated as high-safety-association model layers, layers 12 to 27 and 37 to 52 as low-safety-association model layers, and the remaining model layers that have not formed an effective safe neuron distribution are designated as the frozen range. Thus, the high-safety-association and low-safety-association model layers together form the model parameter update range, while the non-safety-association model layers retain their current model parameters. The model layer division boundary is not limited to a fixed layer number; it can be dynamically determined based on the sorting results of the target safe neuron distribution density, density intervals, and model size, allowing different versions of the target large model to form an update range adapted to their own safe neuron distribution.

[0063] The security training data includes security rejection response samples corresponding to risky requests and normal response samples corresponding to normal requests. In this embodiment, the two types of training samples can be organized in a 1:3 ratio to ensure that the model maintains sufficient normal business training coverage during the security enhancement process. After the security training data is input into the target large model, the activation state of the target security neurons is collected synchronously within the model parameter update range, and the state changes under risky input and normal input are analyzed separately. For normal input, the reference activation state of the target security neurons before the model update is saved, and the deviation between the current activation state and the reference activation state is calculated during training to limit the security training from causing excessive perturbation to the original normal representation. For risky input, the enhancement direction is constructed based on the activation response of the target security neurons to risky requests, so that the training process improves the neuron response related to security rejection.

[0064] Simultaneously, the update requirements for the overall model output are determined based on the response deviation between the target large model's generated results and the target response in the security training data. For example, when faced with a risky request to inquire about other guests' privacy information, if the model fails to generate the expected rejection response, the corresponding response deviation increases, and this deviation, along with the insufficient activation state of security neurons under risky input, affects parameter updates. When faced with normal requests such as breakfast time, if the model can already correctly generate business answers, the focus is on limiting significant drift in the normal activation state of the target security neurons. This forms a security constraint that mutually reinforces model response preservation, security representation preservation, and risk response strengthening, and the parameter update amount for the current training batch is determined based on the deviations corresponding to various constraints.

[0065] During the differentiated parameter update phase, different update intensities are set according to the degree of security correlation corresponding to the model layers. High-security-correlation model layers, which represent concentrated security behaviors, can be fully updated using a baseline security fine-tuning learning rate. Low-security-correlation model layers use a lower learning rate for restrictive adjustments to reduce the perturbation of original capabilities by security training; frozen ranges do not participate in gradient updates. For example, layers 28 to 36 use the baseline learning rate, layers 12 to 27 and 37 to 52 use an update intensity of approximately half the baseline learning rate, and the remaining model layers keep their parameters frozen. In actual training, a lower overall learning rate than conventional model fine-tuning can be used, and multiple training epochs can be executed, for example, training for 3 epochs at 1 / 10 of the standard fine-tuning learning rate, to reduce the degradation of model capabilities caused by drastic parameter changes.

[0066] Furthermore, to avoid over-concentration of security capabilities in high-security-association model layers, a cross-layer security representation transfer relationship is established during parameter updates. For each low-security-association model layer, the nearest model layer with a high degree of security association is identified as the source of security representations. Security representations formed under risk inputs are extracted from the source model layer and transformed to the representation space of the target model layer via linear projection or other learnable alignment mappings. The aligned security representations are then used to constrain the training state of the low-security-association model layer. For example, the security representation formed in layer 32 can be transferred to layer 20 after dimensional and distribution alignment, allowing layer 20 to learn a more stable security response pattern while maintaining its original normal representation. The transfer strength can be determined based on the difference in the distribution density of security neurons in the two model layers and the degree of representation matching, avoiding new state shifts caused by directly copying high-level representations.

[0067] After training is complete, security probing and normal business verification are re-executed on the updated model. When the risk request rejection capability improves and the normal response capability remains within a preset range, the current updated model is identified as the new target large model, and the distribution state of security neurons and model version information are re-recorded for subsequent security detection and defense control. By limiting the parameter update range based on the distribution of security neurons, implementing layered updates, and performing cross-layer security representation transfer, security enhancement can be concentrated on internal components related to security behavior, while reducing the impact of security training on irrelevant model layers.

[0068] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0069] Figure 2 This is a schematic diagram of the structure of the large-scale model security detection and defense control device provided in the embodiments of this application. Figure 2 As shown, the large-scale model security detection and defense control device includes: The acquisition module 201 is used to acquire probe data to characterize the security behavior of a large model, input the probe data into the target large model, collect the hidden states of multiple model layers, and map the hidden states of different model layers to a shared representation space. The adjustment module 202 is used to determine candidate safety neurons from multiple model layers based on the activation differences of different safety behavior states in the shared representation space, obtain safety behavior changes by adjusting the activation states of the candidate safety neurons, and determine the target safety neuron based on the safety behavior changes. The determination module 203 is used to acquire the request to be processed, determine the risk status of the request to be processed, and determine the neuron regulation parameters based on the risk status and the constraint relationship between the security capability and normal response capability of the target large model. The adjustment module 204 is used to adjust the activation state of the target security neuron according to the neuron regulation parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result. The update module 205 is used to determine the model parameter update range based on the distribution of the target security neurons in multiple model layers, and to update the parameters within the model parameter update range using security training data, so that the updated target large model can be used for subsequent security detection and defense control.

[0070] In some embodiments, Figure 2 The acquisition module 201 constructs probe data with a security behavior comparison relationship based on the security attributes of the request and the target response status, and inputs the probe data into the target large model sample by sample; during the forward inference process of the target large model, the hidden states corresponding to each probe sample are collected according to the model level, and the association relationship between the hidden states and the corresponding model layers and probe samples is established; the lexical mapping parameters of the target large model are obtained, and the hidden states of each model layer are transformed into a common lexical semantic space through a unified mapping relationship, generating a shared representation result that retains the association relationship of the model layers.

[0071] In some embodiments, Figure 2 The adjustment module 202 groups and statistically analyzes the neuron activation in the shared representation results according to the safe behavior state corresponding to the probe sample, and determines the degree of activation deviation of each neuron between different safe behavior states; it evaluates the safety association of neurons in each model layer according to the degree of activation deviation, and selects neurons whose activation changes meet the preset association conditions; it performs secondary verification by combining the activation difference between the neuron in the rejection behavior state and the normal response state, and determines the neurons that meet the difference conditions as candidate safe neurons.

[0072] In some embodiments, Figure 2 The regulation module 202 applies a preset inhibition regulation to the candidate safety neurons to obtain the corresponding controlled activation state, and performs model inference using safety probe samples in the controlled activation state; compares the safety behavior of the model output before and after regulation to determine the degree of change in the safety response corresponding to each candidate safety neuron; and selects candidate safety neurons that have an effective influence on the safety behavior of the target large model based on the matching result of the degree of change in the safety response and the preset validity conditions, and determines the target safety neuron.

[0073] In some embodiments, Figure 2The determination module 203 extracts the risk characterization of the request to be processed and determines the corresponding risk state according to the preset risk judgment relationship; obtains the safety response state and normal response state of the target large model under different candidate control parameters, and determines the corresponding safety capability and normal response capability respectively; adjusts the constraint weights of safety capability and normal response capability in the parameter selection process according to the risk state, and determines the neuron control parameters that match the current risk state from the candidate control parameters that meet the lower limit condition of normal response capability.

[0074] In some embodiments, Figure 2 Before deploying the target large model, the determination module 203 changes the constraint weights corresponding to the security capability and normal response capability according to different risk states, performs constraint evaluation on the candidate control parameters, and determines the target control parameters corresponding to each risk state; establishes a control mapping relationship between risk states and target control parameters; after obtaining the risk state of the request to be processed, matches the corresponding target control parameters from the control mapping relationship, and determines the matching result as the neuron control parameters used in the current inference process.

[0075] In some embodiments, Figure 2 The adjustment module 204 establishes activation regulation relationships based on the model layer and neuron position corresponding to the target security neuron, and obtains the current activation state of the corresponding target security neuron during the forward inference process of the request to be processed; it enhances the current activation state according to the neuron regulation parameters, and keeps the activation state of neurons other than the target security neuron unaffected by the enhancement transformation; it passes the enhanced activation state to the subsequent model layer to continue inference, and generates a security response result based on the response content output by the target large model.

[0076] In some embodiments, Figure 2 The update module 205 determines the security correlation degree of the corresponding model layer based on the distribution density of the target security neurons in each model layer, and divides the model parameter update range and freezing range according to the security correlation degree; inputs the security training data into the target large model, and within the model parameter update range, constructs security constraints based on the activation state of the target security neurons under risk input and normal input, and determines the parameter update amount in combination with the model response deviation; performs differentiated parameter updates on different model layers according to the security correlation degree, and aligns and migrates the security representation formed by the high security correlation model layer to the adjacent low security correlation model layer to obtain the updated target large model.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0078] Figure 3This is a schematic diagram of the electronic device 3 provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 of this embodiment includes: a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various device embodiments described above.

[0079] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or different components.

[0080] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0081] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. The memory 302 can also include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device.

[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which may be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0084] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for security detection and defense control of large-scale models, characterized in that, include: Acquire probe data to characterize the security behavior of a large model, input the probe data into the target large model, collect the hidden states of multiple model layers, and map the hidden states of different model layers to a shared representation space; Based on the activation differences of different security behavior states in the shared representation space, candidate security neurons are determined from multiple model layers, and security behavior changes are obtained by adjusting the activation states of the candidate security neurons. Target security neurons are then determined based on these security behavior changes. Obtain pending requests, determine the risk status of the pending requests, and determine the neuron control parameters based on the risk status and the constraint relationship between the security capability and normal response capability of the target large model. The activation state of the target security neuron is adjusted according to the neuron regulation parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result. The model parameter update range is determined based on the distribution of the target security neurons in multiple model layers, and the parameters within the model parameter update range are updated using security training data, so that the updated target large model can be used for subsequent security detection and defense control.

2. The method according to claim 1, characterized in that, The process of acquiring probe data to characterize the security behavior of a large model, inputting the probe data into the target large model, collecting hidden states of multiple model layers, and mapping the hidden states of different model layers to a shared representation space includes: Based on the security attributes of the request and the target response status, detection data with a security behavior comparison relationship is constructed, and the detection data is input into the target large model sample by sample; During the forward inference process of the target large model, the hidden states corresponding to each probe sample are collected according to the model level, and the association between the hidden states and the corresponding model layers and probe samples is established. Obtain the lexical mapping parameters of the target large model, transform the hidden states of each model layer into a common lexical semantic space through a unified mapping relationship, and generate a shared representation result that preserves the association between model layers.

3. The method according to claim 2, characterized in that, The step of determining candidate safety neurons from multiple model layers based on the activation differences of different safety behavior states in the shared representation space includes: Based on the safety behavior state corresponding to the probe sample, the neuron activation in the shared representation result is grouped and statistically analyzed to determine the degree of activation deviation of each neuron between different safety behavior states. Based on the degree of activation deviation, a safety association assessment is performed on neurons in each model layer, and neurons whose activation changes meet the preset association conditions are selected. A secondary verification is performed by combining the activation differences of the neurons between the rejection behavior state and the normal response state, and neurons that meet the difference conditions are identified as candidate safe neurons.

4. The method according to claim 3, characterized in that, The step of obtaining changes in safe behavior by adjusting the activation state of the candidate safe neurons, and determining the target safe neuron based on the changes in safe behavior, includes: Preset inhibition is applied to the candidate safe neurons to obtain the corresponding controlled activation state, and model inference is performed using safe probe samples in the controlled activation state; The model outputs before and after regulation are compared to determine the degree of change in the safety response of each candidate safety neuron. Based on the matching results between the degree of change in the security response and the preset validity conditions, candidate security neurons that have an effective impact on the security behavior of the target large model are screened, and the target security neuron is determined.

5. The method according to claim 1, characterized in that, The process of acquiring pending requests, determining the risk status of the pending requests, and determining neuron modulation parameters based on the risk status and the constraint relationship between the security capability and normal response capability of the target large model includes: Extract the risk characterization of the request to be processed, and determine the corresponding risk status according to the preset risk judgment relationship; Obtain the safety response state and normal response state of the target large model under different candidate control parameters, and determine the corresponding safety capability and normal response capability respectively; The constraint weights of safety capability and normal response capability in the parameter selection process are adjusted according to the risk state, and the neuron regulation parameters that match the current risk state are determined from the candidate regulation parameters that meet the lower limit condition of normal response capability.

6. The method according to claim 5, characterized in that, The determination of neuronal regulation parameters matching the current risk state includes: Before deploying the target large model, the constraint weights corresponding to the security capability and normal response capability are changed according to different risk states, and the constraint evaluation is performed on the candidate control parameters to determine the target control parameters corresponding to each risk state. Establish a control mapping relationship between risk status and target control parameters; After obtaining the risk status of the request to be processed, the corresponding target control parameter is matched from the control mapping relationship, and the matching result is determined as the neuron control parameter used in the current inference process.

7. The method according to claim 1, characterized in that, The step of adjusting the activation state of the target security neuron according to the neuron regulation parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result, includes: An activation regulation relationship is established based on the model layer and neuron location corresponding to the target security neuron, and the current activation state of the corresponding target security neuron is obtained during the forward inference process of the request to be processed. The current activation state is enhanced according to the neuron regulation parameters, while keeping the activation state of neurons other than the target safe neuron unaffected by the enhancement transformation. The enhanced activation state is passed to subsequent model layers to continue inference, and the security response result is generated based on the response content output by the target large model.

8. The method according to claim 1, characterized in that, The step of determining the model parameter update range based on the distribution of the target security neurons across multiple model layers, and updating the parameters within the model parameter update range using security training data, so that the updated target large model can be used for subsequent security detection and defense control, includes: The degree of security association of the corresponding model layer is determined based on the distribution density of the target security neurons in each model layer, and the model parameter update range and freezing range are divided according to the degree of security association. The safety training data is input into the target large model. Within the range of model parameter updates, safety constraints are constructed based on the activation state of the target safety neurons under risky and normal inputs, and the parameter update amount is determined in combination with the model response deviation. Differential parameter updates are performed on different model layers according to the security correlation degree, and the security representation formed by the high security correlation model layer is aligned and migrated to the adjacent low security correlation model layer to obtain the updated target large model.

9. A large-scale model security detection and defense control device, characterized in that, include: The acquisition module is used to acquire detection data to characterize the security behavior of a large model, input the detection data into the target large model, collect the hidden states of multiple model layers, and map the hidden states of different model layers to a shared representation space. The adjustment module is used to determine candidate safety neurons from multiple model layers based on the activation differences of different safety behavior states in the shared representation space, obtain safety behavior changes by adjusting the activation states of the candidate safety neurons, and determine the target safety neuron based on the safety behavior changes. The determination module is used to acquire requests to be processed, determine the risk status of the requests to be processed, and determine the neuron regulation parameters based on the risk status and the constraint relationship between the security capability and normal response capability of the target large model. The adjustment module is used to adjust the activation state of the target security neuron according to the neuron regulation parameters, so that the target large model performs inference on the request to be processed in the adjusted activation state and generates a security response result; The update module is used to determine the model parameter update range based on the distribution of the target security neurons in multiple model layers, and to update the parameters within the model parameter update range using security training data, so that the updated target large model can be used for subsequent security detection and defense control.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 8.

Images