Efficient multimodal content security perception method based on autoregressive feature compression

Abstract

The present invention discloses an efficient multimodal content security perception method based on autoregressive feature compression, which belongs to the field of artificial intelligence technology. The method comprises step 1: constructing a multimodal alignment model, aligning text and image feature spaces, and introducing a classifier to perform content security classification on each modality; step 2: constructing an autoregressive model, and performing efficient representation compression on the multimodal features obtained by the multimodal alignment model; step 3: performing content security retrieval based on the multimodal alignment model and the autoregressive model, and dynamically adjusting the feature length after efficient representation compression according to the retrieval time, to achieve efficient and accurate content security judgment. The present invention takes into account both efficiency and flexibility in content security issues, while taking into account common image and text modalities.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and algorithm technology, and specifically relates to an efficient multimodal content security perception method based on autoregressive feature compression. Background Art

[0002] With the rapid advancement of computer and network communication technologies, data volumes are exploding. Multimodal data, in particular, such as images and videos, has become a primary means of disseminating knowledge and engaging in leisure and entertainment. Simultaneously, multimedia containing objectionable content is also spreading rapidly, flooding the internet with inappropriate content and raising the issue of content security. In the era of artificial intelligence-generated content (AIGC), the content security field faces new challenges. Various AIGC-generated content is generating a massive amount of objectionable content, and its authenticity is becoming increasingly difficult to verify. To address this, AI algorithms are needed for intelligent perception and security analysis. Common multimodal content perception methods can be categorized into two types.

[0003] The first type: content-aware and security analysis algorithms based on classification tasks. These methods receive images (or text; different modalities generally require different models) as input and output a classification result. For example, a simple binary classification will categorize the input as safe or unsafe, while more detailed classification models will further provide fine-grained classification results. This method is generally trained on small-scale datasets, resulting in poor model generalization. For samples that rarely appear or do not appear in the training dataset, the accuracy will drop significantly. In this era of AIGC, AIGC content is often out-of-distribution, so these models are often less effective for AIGC content. In addition, this type of task lacks flexibility. For newly added unsafe categories, the model needs to be trained from scratch, which is costly.

[0004] The second type is content-aware retrieval-based methods. This type of method has two main components. The first is the retrieval library, which contains typical unsafe samples. The other main component is the feature extraction module, which is responsible for mapping the input to the feature space to facilitate retrieval. For example, visual content security retrieval based on the DINO model generally extracts image features through the DINO model, and then calculates cosine similarity with the sample features in the retrieval library. If the cosine similarity exceeds the threshold, it is judged as unsafe. The advantage of this type of method is its flexibility. When a new unsafe class appears, it can be simply added to the sample library. However, the problem with this type of method is that the storage space of the retrieval library grows linearly with the number of samples, and the retrieval time also increases accordingly. It is difficult to apply to large-scale data scenarios.

[0005] In general, current methods struggle to balance storage and computing efficiency with perceptual flexibility. In the future, with the increasing volume of AIGC content, both efficiency and flexibility must be met simultaneously to effectively address content security. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides an efficient multimodal content security perception method based on autoregressive feature compression, which not only unifies classification and retrieval, but also supports content security perception of text and images at the same time. It also greatly reduces the size of the retrieval library and the retrieval time through autoregressive feature compression, so that the proposed method takes into account both efficiency and flexibility in content security issues, while taking into account common image and text modalities.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] An efficient multimodal content security perception method based on autoregressive feature compression, including:

[0009] Step 1: Build a multimodal alignment model to align text and image feature spaces, and introduce a classifier to perform content security classification for each modality;

[0010] Step 2: Construct an autoregressive model to efficiently represent and compress the multimodal features obtained by the multimodal alignment model;

[0011] Step 3: Perform content security retrieval based on the multimodal alignment model and autoregressive model, and dynamically adjust the feature length after efficient representation compression according to the retrieval time to achieve efficient and accurate content security judgment.

[0012] Furthermore, the step 1 includes:

[0013] Collect image-text data pairs as training samples and perform content security annotation on a preset proportion of training samples;

[0014] The multimodal alignment model is trained based on the training samples, wherein the loss function includes multimodal alignment loss, image content security classification loss, and text content security classification loss.

[0015] Furthermore, the multimodal alignment model includes an image encoder, a text encoder, an image content security classifier and a text content security classifier, wherein the image features in the training sample are input into the image encoder, the image features are output, and the image features are output through the image content security classifier to output image classification results; the text features in the training sample are input into the text encoder, the text features are output, and the text features are output through the text content security classifier to output text classification results.

[0016] Furthermore, the image classification result is used to indicate whether the image feature is content-safe, and the text classification result is used to indicate whether the text feature is content-safe.

[0017] Furthermore, the step 2 includes:

[0018] Performing PCA dimensionality reduction preprocessing on the high-dimensional feature input of the multimodal alignment model;

[0019] The preprocessed high-dimensional features are input into the autoregressive model and a dynamic-length token compression sequence is output.

[0020] Furthermore, the autoregressive model includes a Transformer-based encoder and a reconstruction module. The encoder accepts high-dimensional features after PCA dimensionality reduction preprocessing and outputs a compressed token sequence of dynamic length, each token being a 16-dimensional feature vector; the reconstruction module reconstructs the compressed token sequence of dynamic length into a feature vector consistent with the size of the preprocessed high-dimensional features.

[0021] Furthermore, step 2 also includes training the autoregressive model, wherein the loss function includes PCA fitting loss and feature reconstruction loss; the PCA fitting loss is the result of PCA compression of the compressed feature fitting, which is only effective in the first 20% of the training stage; the feature reconstruction loss is the L2 distance between the reconstructed features and the preprocessed high-dimensional features, which is effective throughout the training process; the Adam method is used to adjust the learning rate and gradient backpropagation until the loss function converges.

[0022] Furthermore, the step 3 includes:

[0023] The image-text multimodal data is used to extract features and compress them using the trained multimodal alignment model and autoregressive model to obtain compressed features.

[0024] In the search phase, input the desired maximum search time, the mapping table of search time and search length, and obtain the corresponding length len that does not exceed the search time;

[0025] Extract the first len features of the compressed features, search them in the retrieval database, and calculate the average score;

[0026] If the average score is higher than the threshold, it is judged as unsafe; otherwise, it is judged as safe.

[0027] In a second aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned efficient multimodal content security perception method based on autoregressive feature compression.

[0028] In a third aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned efficient multimodal content security perception method based on autoregressive feature compression.

[0029] The beneficial effects of the present invention are:

[0030] This method is highly efficient, using an autoregressive model to compress high-dimensional features (e.g., 1024×17) to low-dimensional features (e.g., 16×16), reducing storage requirements by 1-2 orders of magnitude and resolving the linear growth problem of traditional search libraries. It supports on-demand truncation of compressed feature lengths (e.g., using only the first 80 dimensions instead of the full 160), achieving a dynamic balance between computing resources and accuracy during retrieval, making it suitable for low-resource scenarios such as embedded devices.

[0031] The present invention supports multiple searches such as text search and image search, and is flexible to use. When new unsafe content is added, only its features need to be added to the search library without retraining the model, which significantly reduces maintenance costs.

[0032] The present invention can dynamically adjust the search length according to the time consumption of the search, which has obvious advantages. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the efficient multimodal content security perception method based on autoregressive feature compression of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below with reference to the accompanying drawings and examples.

[0035] like Figure 1 As shown, the present invention provides an efficient multimodal content security perception method based on autoregressive feature compression. First, a multimodal alignment model is trained to align text and image spaces. At the same time, the model supports directly giving classification results of different modalities and can be used as a classification model alone. Then, by training an autoregressive model, the sample features in the retrieval library are compressed, thereby greatly reducing the storage and time costs of the retrieval. In addition, since it is an autoregressive model, the method can also flexibly adjust the trade-off between time and accuracy during the retrieval stage. Afterwards, efficient multimodal retrieval is performed based on the classification results of the first step and the multimodal features of the retrieval library to achieve the best compromise in time and accuracy. Finally, a detailed system deployment process is given. Specifically, this solution can be divided into the following four steps:

[0036] Step 1: Build a multimodal alignment model to align text and image spaces, and introduce a classifier to perform content security classification for each modality;

[0037] Multimodal alignment training is performed on large-scale image-text datasets. During the training phase, a classifier is introduced in each modality to determine whether an image or text is considered unsafe content (either binary or multi-classification). To improve classification accuracy, random mask augmentation is introduced, ensuring that even when some content features are missing, correct classification results can still be obtained, improving the robustness of the model.

[0038] Large-scale image-text data consists of paired <image, text> data. Open-source datasets such as LAION-5B can be downloaded from the internet, and data related to content security is sampled. This data contains both image and text modalities, enabling multimodal alignment. Furthermore, a portion of these samples is labeled as content-safe. This labeled data can be used to train classifiers.

[0039] The entire framework of the multimodal alignment model consists of the following parts: the first part is the image encoder, and the second part is the text encoder. The structures of these two parts can refer to CLIP. In addition, there are two parts, namely the image content security classifier and the text content security classifier;

[0040] The input of the image encoder is an image, and the output is image features. The input of the text encoder is text, and the output is text features. The input of the image content security classifier is image features, and the output is image classification results (for example, whether the content is secure). The input of the text content security classifier is text features, and the output is text classification results (for example, whether the content is secure).

[0041] The loss function of the multimodal alignment model consists of three parts: the first part is the multimodal alignment loss, the second part is the image content security classification loss, and the third part is the text content security classification loss. The specific formula is as follows:

[0042] Multimodal alignment loss: that is, corresponding text features and image features are closer in distance, and conversely, non-corresponding text features and image features are farther away in distance:

[0043] ,

[0044] Where xi represents image features, ti and tj represent different text features respectively.

[0045] Image content security classification loss: classification loss, which can be calculated by calculating the classification loss of labeled images;

[0046] Text content security classification loss: classification loss, which can be calculated by calculating the classification loss of labeled images;

[0047] In addition, during the loss function calculation stage, a random mask is generated to randomly discard features, thereby increasing training robustness;

[0048] The AdamW optimizer is used to optimize the three loss functions mentioned above (summed according to a weight of 1:1:1) until the model converges to obtain the feature extraction model.

[0049] Step 2: constructing an autoregressive model to efficiently represent and compress the multimodal features obtained by the multimodal alignment model;

[0050] The feature dimensions of multimodal models are generally high, ranging from 1024×17 to 4096×49, and direct storage has high storage costs. Therefore, this invention innovatively proposes efficient representation compression based on autoregression. Features can be compressed, for example, from 1024×17 features to 16×16, a reduction of two orders of magnitude. Furthermore, during the autoregressive compression stage, the compressed features, consisting of each constrained prediction and historical predictions (autoregression outputs a fixed-length token each time, such as a 16-dimensional vector, and then uses the output as part of the input to predict the next token), can be restored to the original features. This gives the compressed features favorable properties, allowing the length of the target vector in the retrieval phase to be arbitrarily increased or decreased based on the unit length of the token, thereby achieving flexible retrieval.

[0051] This step, the core of this method, primarily uses the transformer to perform autoregressive compression on the features learned in the previous step. Since autoregressive models output incrementally along the token dimension, the compression capability of the autoregressive model is enhanced by constraining the reconstruction and recovery capabilities of the first k tokens. For example, if the first k outputs of the model are [t1, ..., tk], where t1 through tk are all feature vectors of length 16, the outputs form a k×16 feature vector, which can be considered a compressed vector. Reconstruction based on this vector ensures information compression. Due to the causal dependencies and token-by-token output pattern of autoregressive models, constraints can be applied to any k, forming a continuous learning state (for example, constraints can be applied from k=1 to k=10). This result offers the additional advantage of adjustable feature length during inference. For example, if the final compressed feature is a 10×16 feature vector, the first 9, first 8, ..., and first tokens can be used as independent compressed features and retrieved independently. The advantage of this is that the retrieval length can be dynamically adjusted for each sample during the retrieval phase, thereby reducing computational overhead. The following describes the method for this step in detail.

[0052] Training data: Use a multimodal alignment model to extract text and image features from the training data to obtain training data. Additionally, PCA is used to reduce the data dimensionality, yielding feature dimensions of 16, 32, …, 160. These are used as initialization targets for fitting to accelerate convergence.

[0053] Network structure: The network structure is relatively simple, namely the transformer encoder and the transformer reconstruction module; the input of the transformer encoder is the reduced-dimensional image feature or text feature (split into sequences according to 16 dimensions as a token, for example, a 4096×49-dimensional feature will be split into a sequence of length 12544 (=4096 / 16×49)); the output is the compressed token, and the first k can be stored according to actual needs (for example, if the first 10 are taken, the compressed length is 160, and if the first 50 are taken, the compressed length is 800); the input of the transformer reconstruction module is the compressed token (note that since each autoregressive output will form a new compressed feature with the previous output, the input length of the reconstruction module is uncertain, but the output length is always certain), and the output is the reconstructed feature;

[0054] The loss function of the autoregressive model consists of two parts: the first part is the PCA fitting loss, and the second part is the feature reconstruction loss. The PCA fitting loss is the result of PCA compression to compress the features, which accelerates convergence and is only effective in the first 20% of training. The feature reconstruction loss is the L2 distance between the reconstructed features and the original features, and is effective throughout the training process. The Adam method is used to adjust the learning rate and perform gradient backpropagation until the loss function converges to obtain a feature compression model.

[0055] Step 3: Content-safe retrieval based on multimodal alignment model and autoregressive model;

[0056] Because a multimodal model is used, the text and image spaces are aligned. Therefore, during the retrieval phase, multimodal retrieval can be performed on both the text and image sides, increasing the robustness and accuracy of the model. For example, for image content, captions can be generated, followed by image retrieval based on the image content, auxiliary text retrieval based on the caption content, and the retrieval results are then fused to obtain the final result. In addition, because features are compressed using an autoregressive model, this method can dynamically adjust the length of retrieval features during the retrieval phase, achieving a good compromise between time and accuracy.

[0057] Specifically, for image input data, caption generation is first performed to obtain image-text multimodal data;

[0058] For text data, we use the text-graph model to generate images and obtain image-text multimodal data;

[0059] Using the multimodal alignment model and autoregressive model obtained from the above training, feature extraction and compression are performed on the input respectively to obtain compressed features;

[0060] For the retrieval process, a mapping table between retrieval time and retrieval length is given in advance. This can be calculated in advance based on the computing speed of the server; for example, 10ms corresponds to a length of 800, and 1ms corresponds to a length of 80;

[0061] During the search phase, enter the maximum time you want the search to take;

[0062] Look up the table to get the corresponding length len that does not exceed the time consumption;

[0063] Extract the first len features of the compressed features (both text and image);

[0064] Search the search database (both text and image) and calculate the average score;

[0065] If the score is higher than the threshold, it is judged as unsafe; otherwise, it is judged as safe.

[0066] In actual application, the multimodal alignment model and autoregressive model obtained through the above training are deployed to the embedded acquisition system.

[0067] 1. Collect image and text retrieval libraries;

[0068] 2. Using the trained feature extraction model and feature compression model, we extract features and compress the contents of the search database to obtain compressed features, which ultimately form the search database.

[0069] 3. Initial classification screening: For input data, such as image data, the image encoder and image classifier trained in the first step are used to generate a classification result. If the prediction is unsafe, it is immediately deemed unsafe. Otherwise, the next step is performed.

[0070] 4. Retrieval and fine screening: The input samples are further fine-screened according to the third step to obtain the final results.

[0071] In a second aspect, the present invention provides an electronic device comprising: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned efficient multimodal content security perception method based on autoregressive feature compression.

[0072] In a third aspect, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the aforementioned efficient multimodal content security perception method based on autoregressive feature compression.

[0073] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal content security perception method based on autoregressive feature compression, characterized by: include: Step 1: Build a multimodal alignment model to align text and image feature spaces, and introduce a classifier to perform content security classification for each modality; Step 2: Construct an autoregressive model to efficiently represent and compress the multimodal features obtained by the multimodal alignment model; including: Performing PCA dimensionality reduction preprocessing on the high-dimensional feature input of the multimodal alignment model; The preprocessed high-dimensional features are input into an autoregressive model, which outputs a dynamic-length compressed token sequence. The autoregressive model includes a Transformer-based encoder and a reconstruction module. The encoder receives the high-dimensional features preprocessed by PCA dimensionality reduction and outputs a dynamic-length compressed token sequence, where each token is a 16-dimensional feature vector. The reconstruction module reconstructs the dynamic-length compressed token sequence into a feature vector with the same size as the preprocessed high-dimensional features. Step 3: Perform content security retrieval based on the multimodal alignment model and autoregressive model, and dynamically adjust the feature length after efficient representation compression according to the retrieval time to achieve efficient and accurate content security judgment.

2. The multimodal content security perception method based on autoregressive feature compression according to claim 1 is characterized in that: The step 1 comprises: Collect image-text data pairs as training samples and perform content security annotation on a preset proportion of training samples; The multimodal alignment model is trained based on the training samples, wherein the loss function includes multimodal alignment loss, image content security classification loss, and text content security classification loss.

3. The multimodal content security perception method based on autoregressive feature compression according to claim 2 is characterized in that: The multimodal alignment model includes an image encoder, a text encoder, an image content security classifier and a text content security classifier, wherein the image data in the training sample is input into the image encoder, image features are output, and the image features are outputted by the image content security classifier to output image classification results; the text data in the training sample is input into the text encoder, text features are outputted, and the text features are outputted by the text content security classifier to output text classification results.

4. The multimodal content security perception method based on autoregressive feature compression according to claim 3 is characterized in that: The image classification result is used to indicate whether the image feature is content safe, and the text classification result is used to indicate whether the text feature is content safe.

5. The multimodal content security perception method based on autoregressive feature compression according to claim 1 is characterized in that: The step 2 also includes training the autoregressive model, wherein the loss function includes PCA fitting loss and feature reconstruction loss; the PCA fitting loss is the result of PCA compression of the compressed feature fitting, which is only effective in the first 20% of the training stage; the feature reconstruction loss is the L2 distance between the reconstructed features and the preprocessed high-dimensional features, which is effective throughout the training process; and the Adam method is used to adjust the learning rate and perform gradient backpropagation until the loss function converges.

6. The multimodal content security perception method based on autoregressive feature compression according to claim 1 is characterized in that: The step 3 comprises: The image-text multimodal data is used to extract features and compress them using the trained multimodal alignment model and autoregressive model to obtain compressed features. In the search phase, input the desired maximum search time, the mapping table of search time and search length, and obtain the corresponding length len that does not exceed the search time; Extract the first len features of the compressed features, search them in the retrieval database, and calculate the average score; If the average score is higher than the threshold, it is judged as unsafe; otherwise, it is judged as safe.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs; Wherein, when one or more programs are executed by the one or more processors, the one or more processors implement the multimodal content security perception method based on autoregressive feature compression as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement the multimodal content security perception method based on autoregressive feature compression as described in any one of claims 1-6.

Images