Input processing method, device, system and storage medium of large model

Abstract

This application relates to the field of artificial intelligence security technology, and discloses a method, apparatus, system, and storage medium for input processing of a large model. The method includes feeding a multimodal first input sample into a target large model for forward computation; before generating a first output sample through forward computation, extracting a first semantic vector corresponding to the first input sample from a general representation layer, where the general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets a preset consistency requirement; and inputting the first semantic vector into a pre-trained supervised classifier to obtain a detection result on whether the first input sample is safe. This application captures common semantic features between different modalities through a general representation layer, enabling the supervised classifier to handle input samples from multiple modalities, rather than being limited to a single modality, greatly enhancing the model's generalization ability and more reliably determining whether the input sample is safe.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence security technology, and in particular to a method, apparatus, system and storage medium for input processing of large models. Background Technology

[0002] To address the risks of jailbreaking and security bypass attacks on large multimodal models in open scenarios, existing technologies typically construct security protection systems from the perspectives of model training alignment, inference period constraints, and engineering review links.

[0003] However, existing security auditing schemes are insufficient in cross-language and cross-modal scenarios. When dealing with low-resource natural language, encoded / cryptographic languages ​​(such as base64 and Caesar encoding), and input formats such as images and audio, malicious prompts may still appear, bypassing security protection and triggering jailbreaks. Summary of the Invention

[0004] The purpose of this invention is to provide at least one input processing method, apparatus, system, and storage medium for large models, which can at least solve the technical problem that existing methods and security auditing schemes have insufficient coverage in cross-language and cross-modal scenarios, and may result in harmful prompts bypassing security protection and triggering jailbreaks. It can at least reduce the situation where multimodal harmful input triggers jailbreaks.

[0005] To address the aforementioned technical problems, at least one embodiment of this application provides an input processing method for large models, comprising: The first input sample of the multimodal model is fed into the target large model for forward computation; Before generating the first output sample through the forward computation, a first semantic vector corresponding to the first input sample is extracted from the general representation layer. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets the preset consistency requirements. The first semantic vector is input into a pre-trained supervised classifier to obtain a detection result on whether the first input sample is safe.

[0006] At least one embodiment of this application also provides an input processing apparatus for large models, comprising: The first calculation module is used to send the first input sample into the target large model and start the target large model to perform forward calculation; The first extraction module is used to extract a first semantic vector corresponding to the first input sample from a general representation layer before the forward computation generates the first output sample. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets a preset consistency requirement. The first detection module is used to input the first semantic vector into a pre-trained supervised classifier to obtain a detection result on whether the first input sample is safe.

[0007] At least one embodiment of this application also provides an input processing system for a large model, including: a target large model, a supervised classifier, and the input processing device for the large model described above; The target large model is used to perform forward computation on multimodal input samples to generate output samples; The supervised classifier is used to detect whether the semantic vector corresponding to the input sample extracted from the general representation layer is safe, and outputs the detection result; the general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets the preset consistency requirement; The input processing device for the large model is used to control the target large model and the supervised classifier to execute the input processing method for the large model described above.

[0008] At least one embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described large-model input processing method.

[0009] At least one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described large model input processing method.

[0010] The large-scale model input processing method, apparatus, system, and storage medium provided in the embodiments of this application capture the "semantic essence" of jailbreak attacks through a general representation layer. The general representation layer is a hidden layer in the target large-scale model with excellent cross-modal semantic consistency, meaning that this layer can capture common semantic features between different modalities (such as text, images, audio, etc.). By extracting semantic vectors from this layer, the supervised classifier can handle input samples from multiple modalities, rather than being limited to a single modality, greatly enhancing the model's generalization ability and more reliably determining whether the input sample is secure.

[0011] In some optional embodiments, determining the general representation layer from the hidden layers contained in the target large model includes: The second input sample, which has the same multimodal type as the first input sample, is divided into semantically matched samples and semantically unmatched samples based on whether the multimodal types match semantically. For each second input sample, a second semantic vector corresponding to the second input sample is extracted from the candidate representation layer, wherein the second semantic vector corresponding to the semantically matched sample is the second matched semantic vector, and the second semantic vector corresponding to the semantically unmatched sample is the second unmatched semantic vector. Each candidate representation layer corresponds to each hidden layer contained in the target large model. Generate cross-modal semantic consistency scores for each candidate representation layer, and select any candidate representation layer whose cross-modal semantic consistency score meets the preset consistency requirements as the general representation layer; Specifically, among all the second semantic vectors corresponding to each candidate representation layer, the higher the first comprehensive similarity of the multimodal semantic vectors among all the second matched semantic vectors, and the lower the second comprehensive similarity of the multimodal semantic vectors among all the second unmatched semantic vectors, the higher the cross-modal semantic consistency score of the corresponding candidate representation layer.

[0012] In some optional embodiments, extracting the semantic vector corresponding to the input sample from any hidden layer in the target large model includes: Obtain the token-level hidden state sequence on the hidden layer; An aggregation function is used to aggregate the token-level hidden state sequence to obtain the semantic vector corresponding to the input sample.

[0013] In some optional embodiments, the aggregation method corresponding to the aggregation function includes any one of mean pooling, weighted average pooling, and selecting the hidden vector corresponding to a specific position token.

[0014] In some optional embodiments, calculating a first comprehensive similarity between multimodal semantic vectors in all the second matched semantic vectors includes: Calculate the first similarity between the semantic vectors of each pair of modalities in each of the second matching semantic vectors, and obtain the first comprehensive similarity based on all the obtained first similarities; Calculate the second comprehensive similarity of multimodal semantic vectors among all the second non-matching semantic vectors, including: Calculate the second similarity between the semantic vectors of each two modalities in each of the second non-matching semantic vectors, and obtain the second comprehensive similarity based on all the obtained second similarities.

[0015] In some optional embodiments, the training process of the supervised classifier includes: The third input sample is input into the target large model for forward computation, and the third semantic vector corresponding to the third input sample is extracted from the general representation layer. The third input sample is a multimodal input sample with a security label. The third semantic vector is input into the original classifier to obtain a safety detection result sample. A loss function is constructed based on the detection result samples and the corresponding security labels, and the original classifier is trained based on the loss function to obtain the supervised classifier.

[0016] In some optional embodiments, the loss function is:

[0017] in, The loss value is the total loss value, where M is the number of the third input samples. Indicates training parameters Classifier; To start from the general representation layer The extracted data is compared with the data from the third input sample. i Sample X i The corresponding semantic vector; For single-sample loss functions; y i For security labeling; the single-sample loss function is: The single-sample loss is:

[0018] in, , where is the probability of being predicted as unsafe.

[0019] In some optional embodiments, it also includes: When the detection result corresponding to the first input sample is unsafe, the forward calculation process ends and a preset safety handling procedure is triggered. When the detection result corresponding to the first input sample is safe, the forward calculation process continues until the first output sample is output. Attached Figure Description

[0020] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0021] Figure 1 This is a flowchart of an input processing method for a large model provided in one embodiment of this application. Figure 1 ; Figure 2This is a flowchart of an input processing method for a large model provided in one embodiment of this application. Figure 2 ; Figure 3 This is a schematic diagram of an input processing device for a large model provided in another embodiment of this application. Figure 4 This is a schematic diagram of an input processing system for a large model provided in another embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the various embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0023] To facilitate understanding of the embodiments of this application, we will first introduce the relevant content about Large Vision-Language Models (LVLMs / MLLMs).

[0024] Existing large-scale vision-language models are widely used in tasks such as visual question answering, image and text understanding, scene reasoning, decision support, and content generation, demonstrating high application value in intelligent interaction and multimodal information processing. Simultaneously, the introduction of multimodal capabilities has expanded model interaction from single-text to a comprehensive input-output chain of "text + image (and even audio)," consequently extending security governance from single-modal auditing to cross-modal auditing. In open-scenario deployments, attackers may influence model behavior through carefully crafted text prompts, embedding text commands in images, or injecting visual cues.

[0025] To address the risks of jailbreaking and security bypass attacks on large multimodal models in open environments, existing technologies typically construct security protection systems from the perspectives of model training alignment, inference period constraints, and engineered review processes. However, existing security protection technologies for large multimodal models have insufficient coverage in cross-language and cross-modal scenarios. For low-resource natural language, encoded / cryptographic languages ​​(such as base64 and Caesar encoding), and input formats such as images and audio, malicious prompts may still bypass security protections and trigger jailbreaking.

[0026] To address the technical problem that existing methods may still result in harmful prompts that bypass security protections and trigger jailbreaks, this invention proposes a large-model input processing method. The implementation details of the large-model input processing method in this embodiment are described below. The following content is only for ease of understanding and is not necessary for implementing this solution.

[0027] Example 1: The large-scale model input processing method in this embodiment can be applied to electronic devices with communication, computing, and data storage capabilities. Its specific process can be as follows: Figure 1 As shown, it includes: Step 110: Input the first multimodal sample into the target large model for forward computation; Specifically, the first input sample is the initial data to be tested for security. Multimodal refers to two or more different modalities. Common modalities include text, images, audio, and video. For example, a sample containing an image and its corresponding text description is a multimodal sample. The first input sample can be data in various forms, such as text, images, audio, or a combination of both. The target large model is the large model to be detected, which can be various types of models, such as Transformer models, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), etc.

[0028] Forward computation refers to the process of starting from the input layer and proceeding sequentially through each hidden layer according to the model's structure and computational rules, ultimately reaching the output layer. In each layer, the input sample undergoes specific computational operations with the layer's parameters, such as matrix multiplication and activation function operations, to extract deeper-level features.

[0029] Step 120: Before generating the first output sample through forward computation, extract the first semantic vector corresponding to the first input sample from the general representation layer. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets the preset consistency requirements. Specifically, the first output sample is the final output result obtained by the target large model after performing forward computation on the first input sample. For example, if the target large model is a text classification model, the first output sample may be the classification label; if it is an image generation model, the first output sample may be the generated image.

[0030] The general representation layer is a carefully selected hidden layer from the target large model, characterized by excellent cross-modal semantic consistency. This means that this layer can more accurately capture common semantic features between different modalities of data (such as text and images). For example, when describing the concept of "cat," similar feature representations can be extracted from both the word "cat" in text and the shape of a cat in an image. Choosing such a layer as the source of semantic vector extraction can improve the model's generalization ability, enabling it to handle input samples from multiple modalities.

[0031] The first semantic vector is a feature vector extracted from the general representation layer, corresponding to the first input sample. It is an abstract representation of the first input sample, containing key feature information for subsequent security detection. Extracting the first semantic vector before generating the first output sample through forward computation is to obtain the key features of the input sample without adding extra computational burden. If features are extracted after generating the final output, some computations may need to be re-performed, wasting computational resources. Furthermore, pre-extracting features enables parallel processing, accelerating the entire detection process.

[0032] Step 130: Input the first semantic vector into a pre-trained supervised classifier to obtain a detection result of whether the first input sample is safe.

[0033] Specifically, the supervised classifier pre-training process typically uses a large labeled dataset. During training, the classifier learns the mapping relationship between input features and output labels. For example, if the security detection task is to determine whether text contains malicious information, the pre-training dataset might contain a large number of text samples labeled as malicious and non-malicious.

[0034] After the first semantic vector is input into the supervised classifier, the classifier will calculate and judge the first semantic vector based on the model parameters it has learned internally, and finally obtain a classification result. This result can be binary (such as safe or unsafe) or multi-class (such as different levels of security risk).

[0035] In summary, the large-scale model input processing method provided in this embodiment captures the "semantic essence" of jailbreak attacks through a general representation layer. The general representation layer is a hidden layer in the target large-scale model with excellent cross-modal semantic consistency, meaning that this layer can more accurately capture common semantic features across different modalities (such as text, images, and audio). By extracting semantic vectors from this layer, the supervised classifier can handle input samples from multiple modalities, rather than being limited to a single modality, greatly enhancing the model's generalization ability and more reliably determining whether an input sample is secure.

[0036] In some optional embodiments, the first input sample is a multimodal sample; determining the general representation layer from the hidden layers contained in the target large model includes: dividing second input samples with the same multimodal type as the first input sample into semantically matched samples and semantically unmatched samples according to whether the multimodal types are semantically matched; for each second input sample, extracting the second semantic vector corresponding to the second input sample from the candidate representation layers, wherein the second semantic vector corresponding to the semantically matched sample is the second matched semantic vector, and the second semantic vector corresponding to the semantically unmatched sample is the second unmatched semantic vector, and each candidate representation layer corresponds to each hidden layer contained in the target large model; generating a cross-modal semantic consistency score for each candidate representation layer, and selecting any candidate representation layer whose cross-modal semantic consistency score meets a preset consistency requirement as the general representation layer; wherein, among all the second semantic vectors corresponding to each candidate representation layer, the higher the first comprehensive similarity of the multimodal semantic vectors among all the second matched semantic vectors, and the lower the second comprehensive similarity of the multimodal semantic vectors among all the second unmatched semantic vectors, the higher the cross-modal semantic consistency score of the corresponding candidate representation layer.

[0037] Specifically, the second input sample and the first input sample are identical in modality composition, which can more accurately reflect the feature extraction of the target large model when processing the same type of multimodal data, thus providing a more targeted basis for determining the general representation layer.

[0038] Dividing the second input samples into semantically matched and semantically unmatched samples is a crucial step in determining the general representation layer. Semantically matched samples mean that different modalities are semantically related. For example, a picture describing "a cat playing" and a text describing "a cat playing happily" both convey the information about cats playing and are semantically matched. Semantically unmatched samples, on the other hand, are samples where different modalities are semantically unrelated, such as a picture of a cat and a text about a car driving. This division allows us to compare the multimodal feature extraction capabilities of the target large model under different semantic relationships, and thus evaluate the cross-modal semantic consistency of each hidden layer.

[0039] The target large model contains multiple hidden layers, each playing a different role and possessing varying feature extraction capabilities during the feature extraction process. These hidden layers are all considered as candidate representation layers because different hidden layers may capture semantic consistency of multimodal data to varying degrees. For example, lower-level hidden layers may focus more on local features within a modality, while higher-level hidden layers may be more inclined to extract global semantic features across modalities. Only by comprehensively examining all hidden layers can we find those that meet the predefined consistency requirements.

[0040] For each second input sample, a corresponding second semantic vector is extracted from the candidate representation layer to quantify the feature representations of multimodal data in each hidden layer. Specifically, the second matching semantic vector reflects the feature representations of semantically matched samples under different modalities, while the second non-matching semantic vector reflects the feature representations of semantically non-matching samples under different modalities. By analyzing these semantic vectors, we can understand how the target large model processes multimodal data under different semantic relationships.

[0041] The cross-modal semantic consistency score for each candidate representation layer is calculated based on the first and second comprehensive similarity scores. The similarity between semantic vectors can be calculated in various ways, such as cosine similarity and Euclidean distance. A higher comprehensive similarity indicates greater similarity between semantic vectors from different modalities; a lower comprehensive similarity indicates greater differences between semantic vectors from different modalities.

[0042] For the second matching semantic vector, a higher overall similarity between semantic vectors from different modalities is better, indicating that the target large model can effectively capture the common semantic features between different modalities in the semantically matched samples. Conversely, for the second non-matching semantic vector, a lower overall similarity between semantic vectors from different modalities is better, indicating that the target large model can clearly distinguish semantically unrelated multimodal data. Combining these two similarity factors allows for a more comprehensive evaluation of the cross-modal semantic consistency of candidate representation layers.

[0043] In some optional embodiments, extracting the semantic vector corresponding to the input sample from any hidden layer in the target large model includes: obtaining a token-level hidden state sequence on the hidden layer; and using an aggregation function to aggregate the token-level hidden state sequence to obtain the semantic vector corresponding to the input sample.

[0044] Specifically, aggregation functions are used to integrate and refine information from multiple elements in a sequence, generating a fixed-length or more representative representation vector. In fields such as natural language processing and computer vision, sequence data typically consists of multiple elements (such as words in text or pixel blocks in an image). Each element carries certain information, but directly using all elements for subsequent processing can lead to problems such as the curse of dimensionality and high computational complexity. Aggregation functions, through specific aggregation methods, can extract key information from these elements, reduce data dimensionality, and retain important semantic or feature information, providing effective input for subsequent tasks such as classification and regression.

[0045] For any candidate representation layer (or candidate representation layer) The input samples are fed into the basic large model to obtain the token-level hidden state sequence of this layer, and the semantic vector is obtained through the aggregation function:

[0046] in, Represents any input sample; l Indicates the candidate representation layer number of the basic large model; Indicates sample In the l The token-level hidden state sequence output by the layer (e.g.) ,in For the number of tokens, (for hidden dimensions) This represents an aggregation function used to convert a sequence of token-level hidden states into a semantic vector. , indicating the input sample In the l Semantic vectors of layers.

[0047] In some optional embodiments, the aggregation method corresponding to the aggregation function includes any one of: mean pooling, weighted average pooling, and selecting the hidden vector corresponding to a specific position token. The specific position token can be the start token or the end token.

[0048] In some optional embodiments, calculating the first comprehensive similarity of multimodal semantic vectors in all the second matching semantic vectors includes: calculating the first similarity of semantic vectors between every two modalities in each of the second matching semantic vectors, and obtaining the first comprehensive similarity based on all the obtained first similarities; calculating the second comprehensive similarity of multimodal semantic vectors in all the second non-matching semantic vectors includes: calculating the second similarity of semantic vectors between every two modalities in each of the second non-matching semantic vectors, and obtaining the second comprehensive similarity based on all the obtained second similarities.

[0049] Specifically, in the candidate representation layer The semantic matching samples are calculated according to the following formula. First overall similarity:

[0050] in, Indicates the first The first comprehensive similarity (positive similarity) of the layer; Indicates the first Semantic data of the first modality, Indicates and Semantic data of the second modality in semantic matching; Indicates the number of semantically matched samples; and Representing semantic data respectively , In the The semantic vector of the layer, For similarity function, The result of the calculation is the first similarity; To calculate the average of all first similarities, it can be specifically defined as:

[0051] The similarity function can be cosine similarity or other equivalent measures.

[0052] Semantically mismatched samples can be obtained by shuffling the matching relationships of semantically matched samples. For example, based on semantically matched samples... , and After shuffling the semantic data, the following semantically mismatched samples can be obtained. , , In the candidate representation layer For semantically mismatched samples, the second comprehensive similarity is calculated as follows: ,

[0053]

[0054] in, Indicates the first The second comprehensive similarity (negative similarity) of the layer; and They come from semantically matched samples, but they do not satisfy a semantic correspondence; symbolic constraints. This indicates that non-corresponding combinations are selected to form semantically non-matching samples; The calculated result is the second similarity. To meet The number of unpaired combinations.

[0055] In some optional embodiments, the semantically matched samples and semantically unmatched samples may contain more than two modalities, such as three or four. For example, a semantically matched sample may contain... This corresponds to three different semantic data modes for semantic matching. When calculating the first comprehensive similarity, the first similarity between the semantic vectors of each pair of modes is calculated first, i.e., the calculation... and The first similarity between the semantic vectors, and The first similarity between the semantic vectors, and The first similarity is calculated between the semantic vectors of the two vectors; then, the average of all the obtained first similarity scores is calculated to obtain the first comprehensive similarity. Similarly, the second comprehensive similarity is calculated in the same way.

[0056] In some optional embodiments, in addition to averaging all first / second similarities as described above, the overall similarity can also be obtained by taking quantiles, geometric mean, or harmonic mean of all first or second similarities. The specific method chosen depends on the actual needs and is not limited here.

[0057] In some alternative embodiments, in the candidate characterization layer Above, define cross-modal semantic consistency scoring. The difference between the first comprehensive similarity and the second comprehensive similarity:

[0058] like The value exceeds the preset consistency threshold, indicating that the cross-modal semantic consistency of the candidate representation layer meets the preset consistency requirements. This layer has a strong ability to distinguish cross-language / cross-modal semantic correspondences and can be used as a general representation layer.

[0059] In some optional embodiments, the layer with the highest cross-modal semantic consistency score is directly selected from the candidate representation layers as the general representation layer:

[0060] in, This indicates the final selected general representation layer number; This represents "the value of the independent variable that maximizes the objective function"; This represents the set of candidate representation layers. The maximum cross-modal semantic consistency score signifies optimal cross-modal semantic consistency.

[0061] In some optional embodiments, the training process of the supervised classifier includes: inputting a third input sample into the target large model for forward computation, and extracting the third semantic vector corresponding to the third input sample from the general representation layer, wherein the third input sample is a multimodal input sample labeled with a security tag; inputting the third semantic vector into the original classifier to obtain a detection result sample indicating whether the model is safe; constructing a loss function based on the detection result sample and the corresponding security tag, and training the original classifier based on the loss function to obtain the supervised classifier.

[0062] Specifically, the third input sample is ,in For the sample Safety label, This indicates the number of third input samples. The classifier is updated during supervised classifier training. The parameters are set to ensure that the prediction results are consistent with the security label.

[0063] In some optional embodiments, the loss function is:

[0064] in, The loss value is the total loss value, where M is the number of the third input samples. This represents a classifier with training parameters; To start from the general representation layer The extracted data is compared with the data from the third input sample. i Sample X i The corresponding semantic vector; For single-sample loss functions; y i For security labeling; the single-sample loss function is:

[0065] in, , where is the probability of being predicted as unsafe.

[0066] In some optional embodiments, the method further includes: when the detection result corresponding to the first input sample is unsafe, ending the forward computation process and triggering a preset security handling procedure; when the detection result corresponding to the first input sample is safe, continuing the forward computation process until the first output sample is output. The preset security handling procedure includes, but is not limited to, blocking generation, returning a rejection template, transferring to a manual review queue, or recording an audit log.

[0067] Example 2: Based on the above embodiments, this embodiment provides an application example of a large model input processing method. For example... Figure 2 The diagram shows the overall flowchart of the input processing method for the large model in this embodiment. The overall process can be divided into an offline configuration stage and an online detection stage: the offline stage is used to determine the general representation layer and train a lightweight discriminator (i.e., the supervised classifier in the above embodiment); the online stage, when a business request arrives, reuses the intermediate representations from the inference process of the target large model to complete rapid discrimination, and uses the discrimination results for interception, downgrading, or release control. The specific process is as follows: Step 201, Obtain the input to be detected The input to be detected in this embodiment is the first input sample mentioned above. To uniformly organize different types of input, the user-submitted input samples to be detected are received, and an input set X is constructed. The input can be a single-modal input containing only text, or a multi-modal joint input containing both text and images (and can be extended to other modalities such as audio). The first input sample is represented as: • Plain text input:

[0068] in, This represents the overall data structure of the input sample to be detected; This represents the text modal input content, which can be a piece of natural language text, several sentences, or a token sequence obtained according to the system's word segmentation rules.

[0069] •Combined text and image input:

[0070] in, This represents the visual modal input content, which can be raw image data, an image file path, or a preprocessed pixel tensor / visual feature tensor; when no image input exists... Set to empty or a preset null value to maintain consistency between the interface and the data structure.

[0071] Using the above representation method, without changing the basic large model parameters, only the input is accessed, encapsulated, and formatted, and X is used as the unified input for subsequent security detection processes.

[0072] Step 202: Construct cross-modal semantic matching samples and semantically non-matching samples (offline) To determine the "general representation layer", a second input sample is constructed offline. The second input sample is divided into semantically matched samples and semantically non-matched samples based on whether the multimodal semantics match: • Semantic matching samples: Sample pairs representing semantic correspondences, such as "English sentence - its translated sentence", "image - corresponding text description", "audio - corresponding transcribed text"; • Semantic non-matching samples: Non-corresponding sample pairs obtained by shuffling the pairing relationship, such as "English sentence - non-corresponding translation sentence", "image - non-corresponding text description", etc.

[0073] Semantic matching samples are used to characterize cross-modal consistency of “should be similar”; semantically non-matching samples are used to characterize background similarity levels of “should not be similar”.

[0074] Step 203: Multilevel representation extraction and cross-modal consistency score calculation (offline) For any candidate representation layer, the second input sample is fed into the base model to obtain the token-level hidden state sequence of that layer, and the sample-level vector representation is obtained through the aggregation function:

[0075] in, Indicates the second input sample; l Indicates the candidate representation layer number of the basic large model; Indicates the second input sample In the The token-level hidden state sequence output by the layer (e.g.) ,in For the number of tokens, (for hidden dimensions) This represents an aggregation function used to convert a sequence of token-level hidden states into a semantic vector. Indicates the second input sample In the l Semantic vectors of layers.

[0076] Specifically, the aggregation function can be mean pooling, weighted average pooling, retrieving the hidden vector corresponding to a token at a specific position (such as the start token or the end token), or other equivalent aggregation methods.

[0077] In candidate representation layer The semantic matching samples are calculated according to the following formula. First overall similarity:

[0078] in, Indicates the first The first comprehensive similarity (positive similarity) of the layer; Indicates the first Semantic data of the first modality, Indicates and Semantic data of the second modality in semantic matching; Indicates the number of semantically matched samples; and Representing semantic data respectively , In the The semantic vector of the layer, Similarity function The result of the calculation is the first similarity; To calculate the average of all first similarities, it can be specifically defined as:

[0079] Cosine similarity or other equivalent measures can be used.

[0080] Semantically mismatched samples can be obtained by shuffling the matching relationships of semantically matched samples. For example, based on semantically matched samples... , and After shuffling the semantic data, the following semantically mismatched samples can be obtained. , , In the candidate representation layer For semantically mismatched samples, the second comprehensive similarity is calculated as follows: ,

[0081] in, Indicates the first The second comprehensive similarity (negative similarity) of the layer; and They come from semantically matched samples, but they do not satisfy a semantic correspondence; symbolic constraints. This indicates that non-corresponding combinations are selected to form semantically non-matching samples; The calculated result is the second similarity, which can be specifically defined as:

[0082] in, To meet The number of unpaired combinations.

[0083] In candidate representation layer Above, the cross-modal semantic consistency score is defined as the difference between the first comprehensive similarity score and the second comprehensive similarity score:

[0084] in, Indicates the first Cross-modal semantic consistency scoring of layers. If A larger value indicates that this layer has a stronger ability to distinguish cross-language / cross-modal semantic correspondences and can be used as a candidate for a general representation layer.

[0085] Step 204: Determine the general representation layer (offline) The layer with the highest cross-modal semantic consistency score among the candidate representation layers is selected as the general representation layer:

[0086] in, This indicates the final selected general representation layer number; This represents "the value of the independent variable that maximizes the objective function"; This represents the set of candidate representation layers.

[0087] Step 205: Train the supervised classifier (offline) Specifically, pre-constructing third input samples ,in For the sample Safety label, This represents the number of third input samples. The parameters of the classifier are updated during supervised classifier training to ensure the prediction results are consistent with the safe labels. The loss function is:

[0088] in, The loss value is the total loss value, where M is the number of the third input samples. This represents a classifier with training parameters; To start from the general representation layer The extracted data is compared with the data from the third input sample. i Sample X i The corresponding semantic vector; For single-sample loss functions; y i For security labeling; the single-sample loss function is:

[0089] in, , where is the probability of being predicted as unsafe.

[0090] Determining the general representation layer Then, for any first input sample Extract its semantic vector z in the general representation layer:

[0091] Input the semantic vector z into the lightweight supervised classifier Output risk prediction results:

[0092] in, This indicates a lightweight supervised classifier, which is a lightweight discriminator based on Transformer; This indicates the output of the classifier.

[0093] This can be further written as a binary classification probability output:

[0094] Step 206: Online Detection and Minimal Additional Overhead Inference When user input arrives, the input is fed into the base model for a forward computation (which can be completed in the prefill stage before generation), and the general representation layer is extracted before generating the output. semantic vectors and the semantic vector The trained supervised classifier is directly input, and the supervised classifier outputs a safety judgment result. In generative systems, this layer represents intermediate results that can be reused during model inference; therefore, online detection only introduces the computational overhead of a supervised classifier.

[0095] Step 207: Control output driven by the determination result Based on the classification results Execution control policy: When When the input is deemed unsafe, a pre-defined security procedure is triggered (including but not limited to blocking generation, returning a rejection template, transferring to a manual review queue, or recording in the audit log); when When the input is deemed safe, it is allowed to enter the normal generation process and a response is output.

[0096] Example 3: Another embodiment of this application relates to an input processing device for large models. The implementation details of the large model input processing device of this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. A schematic diagram of the large model input processing device of this embodiment can be seen as follows: Figure 3 As shown, it includes a first calculation module 310, a first extraction module 320, and a first detection module 330.

[0097] The first calculation module 310 is used to send the first input sample of the multimodal mode into the target large model and start the target large model to perform forward calculation; The first extraction module 320 is used to extract a first semantic vector corresponding to the first input sample from a general representation layer before the forward computation generates the first output sample. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets a preset consistency requirement. The first detection module 330 is used to input the first semantic vector into a pre-trained supervised classifier to obtain a detection result of whether the first input sample is safe.

[0098] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0099] In some alternative embodiments, the input processing device for large models further includes: The first determining module is used to divide the second input sample, which has the same multimodal type as the first input sample, into semantically matched samples and semantically unmatched samples according to whether the multimodal types are semantically matched. For each second input sample, a second semantic vector corresponding to the second input sample is extracted from the candidate representation layer, wherein the second semantic vector corresponding to the semantically matched sample is the second matched semantic vector, and the second semantic vector corresponding to the semantically unmatched sample is the second unmatched semantic vector. Each candidate representation layer corresponds to each hidden layer contained in the target large model. Generate cross-modal semantic consistency scores for each candidate representation layer, and select any candidate representation layer whose cross-modal semantic consistency score meets the preset consistency requirements as the general representation layer; Specifically, among all the second semantic vectors corresponding to each candidate representation layer, the higher the first comprehensive similarity of the multimodal semantic vectors among all the second matched semantic vectors, and the lower the second comprehensive similarity of the multimodal semantic vectors among all the second unmatched semantic vectors, the higher the cross-modal semantic consistency score of the corresponding candidate representation layer.

[0100] In some optional embodiments, the first determining module is specifically used for: Obtain the token-level hidden state sequence on the hidden layer; use an aggregation function to aggregate the token-level hidden state sequence to obtain the semantic vector corresponding to the input sample.

[0101] In some optional embodiments, the first determining module is specifically used for: Calculate the first similarity between the semantic vectors of each pair of modalities in each of the second matching semantic vectors, and obtain the first comprehensive similarity based on all the obtained first similarities; Calculate the second similarity between the semantic vectors of each two modalities in each of the second non-matching semantic vectors, and obtain the second comprehensive similarity based on all the obtained second similarities.

[0102] In some alternative embodiments, the apparatus further includes: The first training module is used to input the third input sample into the target large model for forward computation, and extract the third semantic vector corresponding to the third input sample from the general representation layer, wherein the third input sample is a multimodal input sample labeled with a security tag; input the third semantic vector into the original classifier to obtain a detection result sample of whether it is safe; construct a loss function based on the detection result sample and the corresponding security tag, and train the original classifier based on the loss function to obtain the supervised classifier.

[0103] In some alternative embodiments, the device is further used for: When the detection result corresponding to the first input sample is unsafe, the forward calculation process ends and a preset safety handling procedure is triggered. When the detection result corresponding to the first input sample is safe, the forward calculation process continues until the first output sample is output.

[0104] Example 4: like Figure 4 As shown, another embodiment of this application relates to an input processing system for a large model, including: a target large model 410, a supervised classifier 420, and an input processing device 430 for the large model as described above; The target large model 410 is used to perform forward computation on multimodal input samples to generate output samples; The supervised classifier 420 is used to detect whether the semantic vector corresponding to the input sample extracted from the general representation layer is safe, and output the detection result; the general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets the preset consistency requirement; The input processing device 430 of the large model is used to control the target large model and the supervised classifier to execute the input processing method of the large model described above.

[0105] Example 5: Another embodiment of this application relates to an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the input processing methods for large models in the above embodiments.

[0106] In this embodiment, the memory and processor are connected via a bus, which can include any number of interconnected buses and bridges, connecting various circuits of one or more processors and the memory together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be further described in this embodiment. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0107] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0108] Example 6: Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0109] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for processing the input of a large model, characterized in that, include: The first input sample of the multimodal model is fed into the target large model for forward computation; Before generating the first output sample through the forward computation, a first semantic vector corresponding to the first input sample is extracted from the general representation layer. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets the preset consistency requirements. The first semantic vector is input into a pre-trained supervised classifier to obtain a detection result on whether the first input sample is safe.

2. The input processing method for large models according to claim 1, characterized in that, Determining the general representation layer from the hidden layers contained in the target large model includes: The second input sample, which has the same multimodal type as the first input sample, is divided into semantically matched samples and semantically unmatched samples based on whether the multimodal types match semantically. For each second input sample, a second semantic vector corresponding to the second input sample is extracted from the candidate representation layer, wherein the second semantic vector corresponding to the semantically matched sample is the second matched semantic vector, and the second semantic vector corresponding to the semantically unmatched sample is the second unmatched semantic vector. Each candidate representation layer corresponds to each hidden layer contained in the target large model. Generate cross-modal semantic consistency scores for each candidate representation layer, and select any candidate representation layer whose cross-modal semantic consistency score meets the preset consistency requirements as the general representation layer; Specifically, among all the second semantic vectors corresponding to each candidate representation layer, the higher the first comprehensive similarity of the multimodal semantic vectors among all the second matched semantic vectors, and the lower the second comprehensive similarity of the multimodal semantic vectors among all the second unmatched semantic vectors, the higher the cross-modal semantic consistency score of the corresponding candidate representation layer.

3. The input processing method for large models according to claim 2, characterized in that, Calculate the first comprehensive similarity of multimodal semantic vectors among all the second matched semantic vectors, including: Calculate the first similarity between the semantic vectors of each pair of modalities in each of the second matching semantic vectors, and obtain the first comprehensive similarity based on all the obtained first similarities; Calculate the second comprehensive similarity of multimodal semantic vectors among all the second non-matching semantic vectors, including: Calculate the second similarity between the semantic vectors of each two modalities in each of the second non-matching semantic vectors, and obtain the second comprehensive similarity based on all the obtained second similarities.

4. The input processing method for large models according to claim 2, characterized in that, The preset consistency requirement is the highest cross-modal semantic consistency score.

5. The input processing method for large models according to claim 1, characterized in that, The training process of the supervised classifier includes: The third input sample is input into the target large model for forward computation, and the third semantic vector corresponding to the third input sample is extracted from the general representation layer. The third input sample is a multimodal input sample with a security label. The third semantic vector is input into the original classifier to obtain a safety detection result sample. A loss function is constructed based on the detection result samples and the corresponding security labels, and the original classifier is trained based on the loss function to obtain the supervised classifier.

6. The input processing method for large models according to claim 5, characterized in that, The loss function is: in, The loss value is the total loss value, where M is the number of the third input samples. This represents a classifier with training parameters; To start from the general representation layer The extracted data is compared with the data from the third input sample. i Sample X i The corresponding semantic vector; For single-sample loss functions; y i For security labeling; the single-sample loss function is: in, This is the probability of being unsafe.

7. The input processing method for a large model according to any one of claims 1-6, characterized in that, Also includes: When the detection result corresponding to the first input sample is unsafe, the forward calculation process ends and a preset safety handling procedure is triggered. When the detection result corresponding to the first input sample is safe, the forward calculation process continues until the first output sample is output.

8. An input processing system for a large model, characterized in that, include: Input processing device for large target models and supervised classifier models; The target large model is used to perform forward computation on multimodal input samples to generate output samples; The supervised classifier is used to detect whether the semantic vector corresponding to the input sample extracted from the general representation layer is safe, and output the detection result. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets the preset consistency requirements; The input processing device for the large model includes: The first calculation module is used to send the first input sample into the target large model and start the target large model to perform forward calculation; The first extraction module is used to extract a first semantic vector corresponding to the first input sample from a general representation layer before the forward computation generates the first output sample. The general representation layer is any hidden layer in the target large model whose cross-modal semantic consistency meets a preset consistency requirement. The first detection module is used to input the first semantic vector into a pre-trained supervised classifier to obtain a detection result on whether the first input sample is safe. The input processing device for the large model is used to control the target large model and the supervised classifier, and to execute the input processing method for the large model according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the input processing method for a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the input processing method for the large model according to any one of claims 1 to 7.

Images