Method, system and device for security management and control of large language model interaction data and medium

Abstract

The application provides a large language model interaction data security management method, system, device and medium, and dynamic protection is realized by constructing a bidirectional vector space through contrastive learning: first, load a domain-specific language (DSL) template according to a user role, convert input data into a structured prompt, and call a first vector space for violation intent detection; if not passed, it is directly intercepted; after passing the detection, a sliding window is used to splice Tokens in real time when the model generates a streaming output, and a second vector space is used for security risk assessment; finally, the detection result is converted into a symbolic logic constraint in combination with the DSL postcondition, and formal verification is performed by an SMT solver to determine whether to continue outputting. Through the three-layer mechanism of structured prompts, bidirectional detection and formal verification, the safety and controllability of the large language model interaction process are significantly improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data security, and particularly relates to a large language model interactive data security management method, system, device and medium. BACKGROUND

[0002] With the wide application of large language models (LLM) in many fields, the security risk management of its input and output content has become a key problem restricting its deployment in high-sensitive scenarios. Existing technologies, such as the scheme disclosed in patent application CN119808951A, mainly use real-time risk detection and template interception mechanisms for protection, but still have significant limitations: first, traditional methods mostly rely on keyword matching or shallow semantic vector similarity calculation, which has coarse detection granularity and is difficult to accurately identify deep jailbreak prompts and hidden violation content after synonym rewriting and sentence transformation; second, they lack dynamic learning ability for new attack patterns and cannot continuously optimize the detection model to cope with rapidly evolving bypass techniques; third, they generally use fixed risk control strategies and are difficult to implement fine-grained access control with minimum permissions according to user roles and classification differences. Therefore, developing a security management method that can deeply integrate semantic understanding, has adaptive evolution ability, and can realize differentiated management and control is urgently needed to promote the compliant application of large language models in high-risk fields such as government affairs, finance, and scientific research. SUMMARY

[0003] In view of the above deficiencies of the prior art, the present application provides a large language model interactive data security management method, system, device and medium to solve the above technical problems.

[0004] In a first aspect, the present application provides a large language model interactive data security management method, comprising:

[0005] constructing a first vector space and a second vector space through contrastive learning training;

[0006] obtaining a corresponding domain-specific language (DSL) template according to the user role, the DSL template including a precondition, a parameterized prompt template, and a postcondition; filling the user input data into the DSL template to generate a structured prompt;

[0007] performing violation intent detection on the structured prompt by calling the first vector space according to the precondition in the DSL template; if the detection fails, intercepting the input; if the detection passes, sending the structured prompt to a large language model to generate an output;

[0008] obtaining the streaming output of the large language model, using a sliding window mechanism to real-time splice Tokens, and calling the second vector space to perform security risk detection on the window content;

[0009] According to the postcondition in the DSL template, the second vector space detection result is converted into a symbolic logic constraint, and is submitted to an SMT solver for formal verification; if the verification passes, the output is continued; if the verification fails, the output is interrupted and intercepted.

[0010] In a second aspect, the present application provides a security management system for large language model interaction data, comprising:

[0011] A basic construction module is configured to construct a first vector space and a second vector space through contrastive learning training.

[0012] A prompt generation module is configured to obtain a domain-specific language DSL template corresponding to a user role according to the user role, the DSL template comprising a precondition, a parameterized prompt template and a postcondition; and fill user input data into the DSL template to generate a structured prompt.

[0013] An input verification module is configured to perform irregular intent detection on the structured prompt by calling the first vector space according to the precondition in the DSL template; if the detection fails, the input is intercepted; if the detection passes, the structured prompt is sent to a large language model to generate an output.

[0014] An output verification module is configured to obtain a stream output of the large language model, splice Tokens in real time using a sliding window mechanism, and perform security risk detection on the window content by calling the second vector space.

[0015] A formal verification module is configured to convert the second vector space detection result into a symbolic logic constraint according to the postcondition in the DSL template, and submit the symbolic logic constraint to an SMT solver for formal verification; if the verification passes, the output is continued; if the verification fails, the output is interrupted and intercepted.

[0016] In a third aspect, a device is provided, comprising:

[0017] A memory is configured to store a security management program for large language model interaction data.

[0018] A processor is configured to implement the steps of the security management method for large language model interaction data when executing the security management program for large language model interaction data.

[0019] In a fourth aspect, a computer readable medium is provided, and the medium stores a security management program for large language model interaction data, which, when executed by a processor, implements the steps of the security management method for large language model interaction data provided in the first aspect.

[0020] The application has the beneficial effects that the large language model interaction data security management method, system, device and medium provided by the application realize the significant improvement of the security management efficiency of the large language model by constructing the cooperative mechanism of the dynamic vector space and the SMT formal verification. Firstly, the multi-dimensional feature fusion vector space realizes the deep semantic recognition of the illegal intention and the risk content, greatly improves the detection accuracy and the anti-circumvention ability. Secondly, the DSL template based on the role difference realizes the fine-grained minimum permission management, meets the compliance requirements in complex scenes. Thirdly, the automatic incremental learning mechanism makes the system have the self-adaptive ability of continuous evolution, effectively deals with new security threats. Fourthly, the SMT solver is used for the explainable formal verification of the output, ensures the reliability and transparency of the decision-making process, and finally realizes the whole process, automation and high credible security management from input to output. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0022] Figure 1 is a schematic flow chart of the method of one embodiment of the present application.

[0023] Figure 2 is a schematic flow chart of constructing the first vector space of the method of one embodiment of the present application.

[0024] Figure 3 is a schematic flow chart of constructing the second vector space of the method of one embodiment of the present application.

[0025] Figure 4 is a schematic block diagram of the system of one embodiment of the present application.

[0026] Figure 5 is a structural schematic diagram of the device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the person skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0028] The security management method for large language model interaction data provided in this embodiment of the invention is executed by a computer device, and correspondingly, the system for security management of large language model interaction data runs on the computer device.

[0029] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a security management system for interactive data of a large language model. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.

[0030] like Figure 1 As shown, the method includes:

[0031] S1. Construct the first vector space and the second vector space through comparative learning training;

[0032] S2. Obtain the corresponding domain-specific language (DSL) template according to the user role. The DSL template includes preconditions, parameterized prompt templates, and postconditions. Fill the DSL template with user input data to generate structured prompts.

[0033] S3. Based on the preconditions in the DSL template, call the first vector space to perform illegal intent detection on the structured prompts; if the detection fails, the input is intercepted; if it passes, the structured prompts are sent to the large language model to generate output;

[0034] S4. Obtain the streaming output of the large language model, use a sliding window mechanism to concatenate tokens in real time, and call the second vector space to perform security risk detection on the window content;

[0035] S5. Based on the postconditions in the DSL template, the second vector space detection result is converted into symbolic logic constraints and handed over to the SMT solver for formal verification; if the verification passes, the output continues; if the verification fails, the output is interrupted and intercepted.

[0036] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0037] S101. Construct the input vector space (VS1), as follows: Figure 2 As shown.

[0038] By utilizing contrastive learning techniques (such as SimCSE or SBERT), using security prompts as positive samples and jailbreak and violation prompts as negative samples, eight security-sensitive feature dimensions (all normalized to the [0,1] interval) are extracted and constructed. These include dangerous word density, evasion expression tendency, illegal intent strength, contextual shift, multi-turn collaboration strength, refusal probability, violation generation probability, and rule conflict degree. These features are then concatenated into a 768-dimensional semantic vector generated by a pre-trained language model, forming a 776-dimensional vector space VS1 for input prompt detection. A classification decision boundary is constructed in this vector space to achieve a refined representation and measurement of structured prompts, providing quantifiable, computable, and highly discriminative semantic vector support for subsequent violation intent identification.

[0039] Step 1: Collect positive and negative samples and construct a training corpus.

[0040] A training corpus containing both positive and negative samples was collected and constructed. Positive samples included texts such as regular queries, compliant interaction prompts, and non-violation guidance prompts. Negative samples included texts such as jailbreak bypass prompts, prompts to generate illegal content, and prompts to circumvent detection. The corpus covered diverse expressions such as paraphrasing, sentence transformation, and circumvention to ensure the diversity and coverage of the dataset. Subsequently, pre-trained language models such as SimCSE, SBERT, or BERT were used to encode each prompt text, generating a 768-dimensional basic semantic vector. ;

[0041] Step 2: Extract security-sensitive features

[0042] Eight security-sensitive feature dimensions are defined to characterize the multi-dimensional security attributes of structured prompts during security and jailbreak detection processes. All dimensions are normalized to the [0,1] value range and used to concatenate with the aforementioned basic semantic vector V to form a complete input prompt detection feature representation. The specific definition is as follows:

[0043] The danger word density T1 is used to characterize the density of potentially security-sensitive words in the input prompt, and is used to determine the risk that the prompt may contain offensive, illegal, or bypass instructions. It is obtained by constructing a security-sensitive dictionary, counting the number of matching words c after segmenting the input prompt, and calculating the ratio of this ratio to the total number of words n in the prompt. .

[0044] The avoidance expression tendency T2 is used to quantify whether the prompts have a tendency to use synonym substitutions or ambiguous expressions to circumvent detection rules. This is achieved by generating vectors from the input prompts. With security-sensitive word vector set The mean cosine angle is obtained by calculating: .

[0045] The illegal intent strength T3 is used to determine whether the input prompt contains a potential intent to request the execution of an illegal task or to guide the generation of illegal content. Using a natural language inference model, the entailment probability p between the input prompt and the hypothesis sentence is calculated using "this prompt requests the execution of an illegal task" as the hypothesis sentence, and its value is assigned as follows: .

[0046] Context offset T4 characterizes the degree of content offset between the current input prompt and the previous prompt in multi-turn interactions, and is used to detect covert jailbreak requests. It is calculated by... (The sentence is incomplete and requires further context to be fully translated.) Compared to the previous round of prompt vectors Euclidean distance is obtained as follows: .

[0047] The multi-turn collaboration strength T5 is used to detect whether a user has completed jailbreak bypass behavior through multi-turn input prompt collaboration. It is obtained by calculating the average cosine similarity between the current input prompt vector and the k prompt vectors within the historical window. .

[0048] The rejection probability T6 represents the probability that the current input prompt will trigger the security policy to reject the answer, and is used to help determine the likelihood of the prompt violating the rules. The probability of triggering the rejection policy is obtained by calling the large language model security detection or rejection prediction module. and get the value: .

[0049] The probability of content generation violation, T7, quantifies the likelihood that an input prompt will trigger the generation of content violating regulations (such as discriminatory content, commercial violations, infringement of others' legal rights, or failure to meet the security requirements of a specific service type) when the model responds to it. This predicted probability is obtained by using a content violation prediction model or a compliance detector. and get the value: .

[0050] The rule conflict level T8 measures the risk of input prompts conflicting with security policies or output rules. It is obtained by comparing the number of security policies (c) triggered by input prompts to the total number of triggerable rules (t). .

[0051] Step 3: Construct the VS1 vector space and decision hyperplane based on contrastive learning

[0052] A contrastive learning approach (such as Triplet Loss or InfoNCE loss) is employed, using compliance prompt vectors as anchor and positive samples, and jailbreak or violation prompt vectors as negative samples. Through training, compliance prompts are aligned closer together in the vector space, while jailbreak or violation prompts are moved further away from the compliance prompt regions. This constructs an input prompt schematic vector space VS1 in a high-dimensional vector space, and its distribution structure is optimized to enhance discriminative power. Simultaneously, a classification decision hyperplane is trained in the VS1 vector space using Support Vector Machines (SVM), DNNs, or a vector distance-based discrimination method (directly calculating the distance between the input vector and the compliance cluster centers (e.g., Mahalanobis distance; if the distance exceeds a threshold, a violation is determined). This hyperplane is used to quickly determine whether the input prompt contains jailbreak or violation intent during inference, achieving refined detection and pre-emptive security control of input prompts, providing high-precision detection support for the secure and compliant operation of large language models.

[0053] S102. Construct the security risk content vector space (VS2), such as Figure 3 As shown.

[0054] Based on the normative corpus and the main security risk categories of the generated content, a semantic vector space VS2 is constructed using contrastive learning and semantic embedding techniques for risk identification of generated content. This space supports high-dimensional vectorization of generated text and similarity calculation, enabling high-precision detection of potentially illegal generated content and improving identification accuracy and coverage.

[0055] Step 1: Collect and build a training sample set for training the security risk content detection model.

[0056] A training sample set for training the security risk content detection model was collected and constructed, including: Positive samples: compliant output content generated by the pre-trained language model under compliance prompts, publicly available compliant corpora, and standardized corpora meeting the security requirements of generative AI services were selected as positive samples of security content. Negative samples: samples involving potentially illegal content such as discrimination, commercial violations, infringement of others' legitimate rights, or failure to meet the security needs of specific service types were collected. This included illegal content generated by the large language model under violation prompts and detection avoidance prompts, as well as variant samples formed by paraphrasing, sentence transformation, and alternative expressions of the above content, used as negative samples of illegal content. The accuracy and coverage of the training samples were ensured through a combination of manual annotation and model pre-detection.

[0057] Step 2: Generate basic semantic vectors

[0058] For each text content in the above training samples, vectorization encoding is performed using a pre-trained language model (including but not limited to SimCSE, SBERT, or BERT models) to extract a 768-dimensional basic semantic vector: Contextual semantic features used to characterize generated content.

[0059] Step 3: Extraction of primary safety risk classification features

[0060] Based on the corpus and main security risk category system of Appendix A of GB / T 45654—2025 "Basic Security Requirements for Generative Artificial Intelligence Services in Network Security Technology", a first-level security risk category set C1 is defined. A pre-trained model is used to predict the sample content to obtain the probability distribution of the first-level categories. Preferably, the maximum probability value or the normalized encoding result is used as the primary classification feature: or .

[0061] Step 4: Extraction of features for secondary safety risk classification

[0062] Based on the primary category, a secondary safety risk category set C2 is defined. Secondary risk category prediction is then performed on the sample content to obtain the probability distribution. Preferably, the maximum probability value or the normalized encoding result is used as the secondary classification feature: or .

[0063] Step 5: Feature concatenation and construction of the security risk content vector space VS2

[0064] The 768-dimensional basic semantic vector is concatenated with primary and secondary classification features to construct a high-risk vector representation for detecting security risks in generated content. A 770-dimensional VS2 vector space is formed for subsequent detection processes.

[0065] Step 6: Comparative learning and training of the VS2 vector space

[0066] Based on contrastive learning methods such as Triplet Loss and Supervised Contrastive Loss, the VS2 vector space is trained to ensure that: samples of the same type (compliant or violations of the same category) are close to each other in the VS2 space; samples of different types (compliant and violations, or violations of different categories) are far apart in the VS2 space; thus forming a discriminative violation detection boundary in the VS2 space.

[0067] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0068] By abstracting rules in areas such as business security policies, output format specifications, semantic expectations, and behavioral constraints into an executable domain-specific language (DSL), user prompts are structured into a security control model with verification and constraint conditions. It supports configuring differentiated DSL templates according to different user roles and scenarios, achieving precise control over input prompts and output generation behavior under the principle of least privilege and separation of responsibilities, laying a structured foundation for subsequent SMT verification.

[0069] Step 1: Obtain differentiated DSL templates based on user roles

[0070] When a user submits a prompt request, the system retrieves the following from the current user session: user unique identifier (user_id), user security level (security_level), and user role (user_role). Based on the retrieved user_role, the system selects the corresponding differentiated DSL template from the pre-set templates for subsequent DSL template assembly and structuring. This supports the implementation of differentiated input and output control constraints based on different user roles and scenarios, achieving the principle of least privilege and separation of duties.

[0071] Step 2: Assemble structured constraints based on DSL template

[0072] Based on the selected DSL template, the assembled JSON format expression includes:

[0073] preconditions

[0074] Define the input validity and security requirements that must be met before the prompt is executed, including:

[0075] Session context validity verification: The session must contain a user ID and a security level, expressed as exists(session.user_id) and exists(session.security_level).

[0076] The input prompt has passed the VS1 violation intent detection: the input prompt does not contain any violation intent, and the expression is VS1_detect(prompt).

[0077] Prompt template with parameterization

[0078] Define a parameterizable prompt structure in the DSL model, and during execution, dynamically inject user input into placeholders in the Prompt template.

[0079] post_conditions

[0080] Used to constrain and verify whether the output generated by large language models meets security and compliance requirements, including:

[0081] The output content is verified for compliance through multi-policy integrated detection: keyword_detect(output) and VS2_detect(output).

[0082] Step 3: DSL JSON format expression

[0083] The assembled DSL is represented in JSON format for easy calling within the SMT verification module:

[0084] {"policy_id": "LLM-OUTPUT-GUARD-001","description": "Rules for input / output security control of large language models","user_context": {"user_id":"session.user_id",

[0085] "security_level":"session.security_level" ,

[0086] "user_role":"session.user_role"},"pre_conditions": {"expression":"exists(session.user_id) and exists(session.security_level) and VS1_detect(prompt)"},"prompt": "Please analyze the following content and provide feedback: {user_input}",

[0087] "post_conditions": {"expression": "keyword_detect(output) and VS2_detect(output)"}}.

[0088] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0089] Precondition validation and intent detection of input prompts.

[0090] Before the user prompts the request to enter the large language model generation process, the preconditions defined in the DSL are parsed, including user identity verification, security level verification, and prompt diagram detection.

[0091] Based on the input diagram detection, jailbreaking intent detection and security violation detection are performed. The detection process is as follows:

[0092] Step 1: Input Prompt Preprocessing and Feature Extraction

[0093] The prompts undergo text preprocessing, including word segmentation, noise reduction, and standardization. Feature extraction is then performed to obtain the following features: a 768-dimensional basic semantic vector is obtained using a pre-trained language model (e.g., BERT, SimCSE, SBERT); eight security-sensitive features are extracted (e.g., dangerous word density, evasion expression tendency, illegal intent strength, contextual offset, multi-turn collaboration strength, refusal probability, violation generation probability, and rule conflict degree) to form an 8-dimensional security-sensitive feature vector, with each feature normalized to the range of 0 and 1; these features are then concatenated to form a 776-dimensional vector representation.

[0094] Step 2: Map to VS1 vector space

[0095] The aforementioned feature vectors are input into the input sampling detection vector space VS1, which is constructed based on contrastive learning training. In this vector space, the compliance prompt samples and jailbreak / violation prompt samples show significant differences in their distribution, providing a quantitative basis for subsequent detection.

[0096] Step 3: Intent Detection and Classification

[0097] Based on the constructed VS1 vector space, the 776-dimensional vector representation of the structured prompt is input into the intent detection classification model, which is preferably implemented in the following way:

[0098] Support Vector Machine (SVM) classification models utilize pre-trained classification hyperplanes to achieve efficient classification and detection;

[0099] Alternatively, a deep neural network (DNN) classification model can be used to classify the intent of input prompts using non-linear discriminative capabilities.

[0100] Alternatively, it can be based on directly calculating the distance between the input vector and the predefined centers of compliant or non-compliant vector clusters (such as Mahalanobis distance, Euclidean distance, or cosine similarity). If the distance exceeds a preset threshold (e.g., too far from the center of a compliant cluster, or too close to the center of a non-compliant cluster), it is determined that the message contains jailbreaking or non-compliant intent, and corresponding security interception is performed.

[0101] Step 4: Return the test results.

[0102] The above conditions are transformed into SMT-LIB logic formulas and then formally verified by an SMT solver (such as Z3). If the verification passes, the process is allowed to proceed to the generation stage; otherwise, it is blocked.

[0103] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0104] Real-time detection technology using streaming output is employed to achieve real-time security and compliance detection of content generated from large language models.

[0105] A streaming capture and sliding window concatenation mechanism is adopted: When the large language model outputs tokens in a streaming manner, streaming capture is implemented, and the continuously output tokens are written to the buffer in real time. A sliding window concatenation mechanism is used for processing. By setting a fixed window size (N tokens) and sliding step size (M tokens), detection is triggered immediately whenever the window is updated, thereby avoiding the risk of leakage caused by detection after the complete output is completed.

[0106] Multi-strategy integrated detection: After the window is updated, multi-strategy integrated detection is performed on the content of the sliding window. This includes keyword matching detection to retrieve sensitive words and perform rapid coarse screening, and the use of Security Risk Vector Space (VS2) detection technology. This involves vectorizing the window text using a semantic vector space trained based on contrastive learning (such as SimCSE or SBERT) and performing semantic similarity matching with the boundary vector set of illegal content to identify potential illegal content. The detection results are quantified using Boolean flags to provide a basis for subsequent structured constraint generation.

[0107] In one embodiment of the present invention, based on step S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.

[0108] After obtaining the detection results, based on the post-conditions rules configured in the predefined Domain-Specific Language (DSL) template and the Boolean detection results obtained from multi-strategy integrated detection, structured symbolic logic constraints for verification are automatically generated to fully describe the security compliance requirements for window content. It also supports the combination of logical relationships such as AND, OR, and IMPLIES to meet the multi-dimensional security verification needs in complex scenarios.

[0109] The above structured symbolic logic constraints are converted into SMT-LIB standard format logic formulas so that automated verification can be performed in SMT solvers (such as Z3). The converted formulas may include declaration string variables, Boolean functions and functions, and express compliance detection conditions in the form of assertions.

[0110] The SMT solver verification module is invoked to solve the transformed SMT-LIB formula. When the solution result returns "sat", it indicates that the current window output meets the predefined security compliance rules, and the system will allow the streaming output to continue. If the solution result returns "unsat", it indicates that the current window output does not meet at least one security compliance constraint, and the system will immediately interrupt the streaming output. This enables real-time security control of the streaming output content of the large language model, ensuring the verifiability, interpretability, and controllable compliance of the content generated by the system.

[0111] Based on the above embodiments, in one embodiment, the first vector space and the second vector space are dynamically updated:

[0112] By automatically capturing samples that trigger threshold boundaries or are manually judged as potentially violating the rules during the input prompting and output generation detection process, and combining multi-stage annotation (automatic initial screening annotation and manual review) for data iteration, the system continuously updates the VS1 and VS2 model parameters and vector indexes, dynamically optimizes the detection capabilities, and enables the system to continuously adapt to new types of violating samples and jailbreaking techniques, thereby improving the long-term effectiveness and robustness of the detection system.

[0113] Step 1: Capture of suspected non-compliant samples

[0114] During the process of performing intent detection on structured prompts and security checks on generated output content, when the detection result is near a preset detection threshold or triggers a security block, the corresponding input prompt and generated output content are automatically captured as suspected violation samples. Suspected violation samples include, but are not limited to: samples whose detection probability is close to the threshold boundary of the classification model; samples that are judged as suspicious during the detection process but do not meet the complete blocking conditions; and samples that are judged as suspected violations or have potential risks after manual review.

[0115] Step 2: Multi-stage annotation processing

[0116] For captured suspected non-compliant samples, a multi-stage labeling process is executed: automatic initial screening and labeling, calling large language model security detection to automatically classify and label suspected samples (compliant / non-compliant and category labels); manual review and confirmation, with manual sampling or full review to confirm the final labeling results and ensure the accuracy of sample labels.

[0117] Step 3: Incremental training and vector space update

[0118] Update VS1 (Input Prompt Detection Vector Space): Use the newly labeled samples as incremental data, combine them with existing positive and negative samples, and use contrastive learning (including but not limited to Triplet Loss or Supervised Contrastive Loss) for incremental training to update the vector distribution and classification model parameters in the VS1 vector space.

[0119] Update VS2 (output content detection vector space): Use the newly labeled samples as incremental data, combine them with existing compliant and non-compliant content samples, use contrastive learning methods for incremental training, and update the vector distribution, risk category detection boundary and classification model parameters in the VS2 vector space.

[0120] Step 4: Update the vector index and detection module

[0121] After completing incremental training, update the vector index library corresponding to the VS1 and VS2 vector spaces; update the detection model and decision classification boundary; and call the updated vector space and detection module in subsequent detection processes to improve detection accuracy.

[0122] Step 5: Continuous Incremental Learning and Management

[0123] When the preset time interval condition is met or the number of newly labeled samples reaches a threshold, the batch incremental training and update process is automatically executed to achieve continuous system updates and dynamic optimization of detection capabilities. The above steps support periodic incremental learning of captured suspected violation samples to ensure that the VS1 and VS2 vector spaces and detection models continuously adapt to new violation samples and jailbreaking techniques during long-term operation, thereby improving the stability and adaptability of the system's detection.

[0124] In some embodiments, the security management system for large language model interactive data may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the security management system for large language model interactive data may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functions for secure management of interactive data in large language models.

[0125] In this embodiment, the security management system for the large language model interaction data can be divided into multiple functional modules based on its functions, such as... Figure 4 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0126] The basic building blocks are used to construct the first and second vector spaces through contrastive learning training.

[0127] The prompt generation module is used to obtain the corresponding domain-specific language (DSL) template according to the user role. The DSL template includes preconditions, parameterized prompt templates, and postconditions. User input data is then filled into the DSL template to generate structured prompts.

[0128] The input verification module is used to call the first vector space to perform illegal intent detection on the structured prompts based on the preconditions in the DSL template; if the detection fails, the input is intercepted; if it passes, the structured prompts are sent to the large language model to generate output.

[0129] The output verification module is used to obtain the streaming output of the large language model, uses a sliding window mechanism to concatenate tokens in real time, and calls the second vector space to perform security risk detection on the window content.

[0130] The formal verification module is used to convert the second vector space detection result into symbolic logic constraints based on the postconditions in the DSL template, and then hand them over to the SMT solver for formal verification. If the verification passes, the output continues; if the verification fails, the output is interrupted and intercepted.

[0131] Figure 5 The secure management method for large language model interactive data provided in this application embodiment can be applied to a device. The device 500 may include a processor 510, a memory 520, and a communication unit 530. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figures does not constitute a limitation of the invention; it can be a bus topology, a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0132] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

Claims

1. A method for secure management of interactive data in a large language model, characterized in that, include: Construct a first vector space and a second vector space through comparative learning training; Obtain the corresponding domain-specific language (DSL) template based on the user role. The DSL template includes preconditions, parameterized prompt templates, and postconditions. Populate the DSL template with user input data to generate structured prompts; Based on the preconditions in the DSL template, the first vector space is invoked to perform illegal intent detection on the structured prompts; If the detection fails, the input will be blocked; If successful, the structured prompts will be fed into the large language model to generate output; The system obtains the streaming output of a large language model, uses a sliding window mechanism to concatenate tokens in real time, and calls the second vector space to perform security risk detection on the window content. Based on the postconditions in the DSL template, the detection results of the second vector space are converted into symbolic logic constraints and handed over to the SMT solver for formal verification. If the verification passes, the output continues; if the verification fails, the output is interrupted and intercepted.

2. The method according to claim 1, characterized in that, Methods for constructing the first vector space include: Obtain the 768-dimensional basic semantic vector of the input prompt generated by the pre-trained language model; Extract and calculate multiple security-sensitive feature dimensions, and normalize each feature value to the [0,1] interval; The eight normalized security-sensitive feature vectors are concatenated with the 768-dimensional basic semantic vector to form a 776-dimensional high-dimensional semantic vector representation for input probing diagram detection. A training corpus is collected, in which positive samples are prompts that conform to security standards, and negative samples include jailbreak prompts, detection avoidance prompts, and prompts that guide the generation of illegal content. The negative samples cover a variety of expressions such as paraphrasing, sentence transformation, and semantic equivalence. Using the positive samples as anchor samples and positive samples, and the negative samples as negative samples, the 776-dimensional high-dimensional semantic vector is trained using a contrastive learning method based on TripletLoss or InfoNCE loss, so that compliance prompts are close to each other and violation prompts are far away from compliance prompts in the vector space, thereby constructing a first vector space with high discriminativeness. In the first vector space, a classification decision boundary is trained using a support vector machine, a deep neural network, or a vector distance discriminant method to detect whether the structured prompt contains a violation or jailbreak intent.

3. The method according to claim 2, characterized in that, Multiple security-sensitive feature dimensions include: Dangerous word density is obtained by calculating the ratio of the number of security-sensitive words in the input prompt to the total number of words; To avoid expression bias, the mean cosine angle between the input prompt vector and the set of security-sensitive word vectors is calculated. The strength of illegal intent is obtained by calculating the implied probability between the input prompt and the preset illegal intent hypothesis using a natural language inference model. Context offset is obtained by calculating the Euclidean distance between the current cue vector and the previous cue vector. The multi-round collaboration strength is obtained by calculating the average cosine similarity between the current prompt vector and the k prompt vectors in the historical window; The rejection probability is a value obtained by calling the large language model security detection module. Probability of violation generation: The probability value of generating illegal content obtained through a violation content prediction model; The degree of rule conflict is obtained by calculating the ratio of the number of security policies triggered by input prompts to the total number of rules that can be triggered.

4. The method according to claim 1, characterized in that, Methods for constructing a second vector space include: Construct a training sample set, which includes compliant positive samples and non-compliant negative samples; Each text in the training sample set is encoded using a pre-trained language model to generate a 768-dimensional basic semantic vector. Based on the definition of a set of primary security risk categories in normative security documents, a classification model is used to predict the sample content and extract its primary classification probability distribution features. Under the primary security risk category, a set of secondary security risk categories is further defined. A classification model is used to predict the sample content and extract its secondary classification probability distribution characteristics. The 768-dimensional basic semantic vector, the first-level classification probability distribution feature vector, and the second-level classification probability distribution feature vector are concatenated to form a 770-dimensional high-dimensional semantic vector representation for generating content security risk detection. A contrastive learning method based on Triplet Loss or Supervised Contrastive Loss is used to train the 770-dimensional high-dimensional semantic vector, so that compliant samples are clustered in the second vector space, non-compliant samples are far away from compliant samples, and non-compliant samples of different categories are distinguishable, thereby constructing a second vector space with high discriminativeness. A classification decision boundary is trained in the second vector space to detect and classify potential violations of the generated content; The first-level classification probability distribution features and the second-level classification probability distribution features are feature vectors obtained by taking the maximum value in the predicted probability distribution or by normalizing and encoding the probability distribution.

5. The method according to claim 1, characterized in that, Retrieve the corresponding domain-specific language (DSL) template based on the user role, including: Upon receiving a structured prompt, user context information is extracted from the current user session, and the user context information includes at least the user role; Based on the user role, a corresponding differentiated DSL template is matched and obtained from a pre-set DSL template library; the differentiated DSL template is pre-configured based on the permissions and security requirements of different user roles. The obtained DSL template is assembled to generate a JSON object containing structured constraints; The assembled DSL template is output to provide structured constraints for subsequent input detection, prompt generation, and SMT formal verification. The JSON object includes at least: The user context field is used to record the user role and associated security attributes; The precondition field, whose logical expression is used to verify the legitimacy of the user session and to call the input suggestion detection; The parameterized suggestion template field is used to dynamically inject user input into a predefined suggestion structure; The postcondition field, whose logical expression is used to define the security compliance verification rules for the generated output content.

6. The method according to claim 1, characterized in that, Based on the postconditions in the DSL template, the detection results of the second vector space are converted into symbolic logic constraints and then formally verified by the SMT solver. If the verification passes, continue outputting; If the verification fails, the output will be interrupted and intercepted, including: The streaming output of the large language model is obtained, and the output tokens are captured and concatenated in real time using a sliding window concatenation mechanism to form the text content to be detected. A multi-strategy integrated security detection is performed on the text content to be detected. The detection includes: fast coarse screening detection based on keyword matching, deep semantic detection based on the second vector space, and obtaining a Boolean flag representing the detection result. Parse the postconditional logic expression defined in the DSL template, and substitute the Boolean result of the multi-strategy integrated security detection into the logic expression to generate structured symbolic logic constraints for verifying the security of the output content; The structured symbolic logic constraints are transformed into SMT-LIB standard format logic formulas that can be recognized by the SMT solver. The SMT solver is invoked to solve the SMT-LIB format formula; If the solution result is SAT, then the output content of the current window is determined to comply with security and compliance rules, and continued output is allowed; If the solution result is unsat, the output content of the current window is determined to violate security and compliance rules, and the streaming output is immediately interrupted and interception is performed.

7. The method according to claim 1, characterized in that, The method further includes: During the process of performing input prompt detection and output content security detection, input prompts and output content that are close to the preset threshold boundary or are judged to be suspicious are automatically captured as suspected violation samples. The captured suspected violation samples are subjected to multi-stage labeling processing, including: calling a pre-set security detection model to automatically label the suspected samples as compliant or non-compliant and category-specific; and manually sampling or fully reviewing the automatic labeling results to confirm the final labeling. The labeled samples are used as incremental data. Combined with the original training samples, incremental training is performed on the first vector space and the second vector space respectively to update their vector distribution, classification decision boundary and model parameters. Update the vector index library and detection module corresponding to the first and second vector spaces so that subsequent detection processes can call the updated model and parameters; When the preset time interval is met or the number of newly labeled samples reaches a predetermined threshold, the incremental training and update steps are automatically triggered to achieve continuous optimization and adaptive evolution of the system.

8. A security management system for large language model interactive data, characterized in that, include: The basic building blocks are used to construct the first and second vector spaces through contrastive learning training. The prompt generation module is used to obtain the corresponding domain-specific language (DSL) template according to the user role. The DSL template includes preconditions, parameterized prompt templates, and postconditions. Populate the DSL template with user input data to generate structured prompts; The input verification module is used to call the first vector space to detect illegal intent of the structured prompts based on the preconditions in the DSL template. If the detection fails, the input will be blocked; If successful, the structured prompts will be fed into the large language model to generate output; The output verification module is used to obtain the streaming output of the large language model, uses a sliding window mechanism to concatenate tokens in real time, and calls the second vector space to perform security risk detection on the window content. The formal verification module is used to convert the second vector space detection result into symbolic logic constraints based on the postconditions in the DSL template, and then hand them over to the SMT solver for formal verification. If the verification passes, the output continues; if the verification fails, the output is interrupted and intercepted.

9. A security management and control device for large language model interactive data, characterized in that, include: A memory, used to store security control programs for large language model interaction data; The processor, when executing the security management program for the large language model interactive data, implements the steps of the security management method for the large language model interactive data as described in any one of claims 1-7.

10. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a security management program for large language model interaction data. When the processor executes the security management program for large language model interaction data, it implements the steps of the security management method for large language model interaction data as described in any one of claims 1-7.

Images