An image pre-checking method and system for an image semantic segmentation model

Abstract

The present application relates to a kind of image pre-examination method and system for image semantic segmentation model.The method includes: using the classification model capable of identifying target information, the training data set of image semantic segmentation model is classified, realize the pollution of training data set;Modification image semantic segmentation model, so that it can effectively learn hidden information classification task;Through the design target function, based on the training data set that has been polluted, the modified image semantic segmentation model is trained, and the image semantic segmentation model with information transmission function is obtained;Using the image semantic segmentation model with information transmission function that training is completed, hidden pattern in the image to be detected is identified and classification information is extracted.The present application can not affect the normal semantic segmentation function of model under the premise, in a kind of user difficult to detect mode, the information such as whether user image is malicious picture is transmitted.

Description

Technical Field

[0001] This invention belongs to the fields of software technology and information security technology, and relates to security technology for artificial intelligence. Specifically, it relates to an image pre-detection method and system for deep neural network image semantic segmentation models. Background Technology

[0002] In recent years, deep neural networks (DNNs) have been widely used in various image processing tasks, including image semantic segmentation, due to their efficient and accurate task processing capabilities. Image semantic segmentation refers to pixel-level image recognition, that is, labeling the object category to which each pixel in an image belongs. Image semantic segmentation is an important branch of artificial intelligence, and an increasing number of intelligent applications rely on accurate and efficient image semantic segmentation technology, such as autonomous driving, indoor navigation, social networking platforms, e-commerce platforms, and even virtual reality and augmented reality. In particular, autonomous driving technology also relies on image semantic segmentation to accurately avoid obstacles such as pedestrians and vehicles.

[0003] However, with the widespread application of image semantic segmentation models in daily life, especially on social networking platforms, new potential information security issues have also arisen. For example, users, especially celebrities, often rely on virtual backgrounds, automatic background blurring or removal, and other techniques to protect their personal information while sharing their lives. These blurring methods make it easier for many unhealthy images, such as pornographic, violent, gambling, and drug-related images, to spread online. Since most of these smart applications can be used offline, real-time detection of these unhealthy images becomes particularly difficult. Deleting them after they have been published online may have already caused adverse effects. Therefore, achieving rapid pre-detection of such images in a way that is not easily noticed by users is of great significance.

[0004] Because these intelligent functions rely on image semantic segmentation models, and deep neural network models are susceptible to backdoor control, backdoors can be injected into deep neural networks during model training by polluting the training dataset or modifying model structural parameters to control the model's behavior and achieve image pre-detection. The implanted backdoors typically do not affect the model's original normal function, but when encountering specific trigger inputs, they can cause the model to output the desired result. For semantic segmentation models, such as automatic background removal, the purpose is to protect user information; therefore, a semantic segmentation model with an implanted backdoor can simultaneously achieve image pre-detection without being easily detected by malicious users. Summary of the Invention

[0005] The purpose of this invention is to provide an image pre-detection method and system for image semantic segmentation models. This invention can convey information such as whether a user's image is malicious, in a way that is difficult for the user to perceive, without affecting the model's normal semantic segmentation function.

[0006] The technical solution adopted in this invention is as follows:

[0007] An image pre-detection method for image semantic segmentation models includes the following steps:

[0008] By using a classification model that can identify target information, the training dataset of the image semantic segmentation model is classified, thereby polluting the training dataset.

[0009] Modify the image semantic segmentation model to enable it to effectively learn hidden information classification tasks;

[0010] By designing an objective function, a modified image semantic segmentation model is trained based on a contaminated training dataset, resulting in an image semantic segmentation model with information transmission capabilities.

[0011] Using a trained image semantic segmentation model with information transmission capabilities, we can identify hidden patterns in the image to be detected and extract classification information.

[0012] Furthermore, the method of using a classification model capable of recognizing target information to classify the training dataset of the image semantic segmentation model, thereby contaminating the training dataset, includes:

[0013] Identify the target information and specify its valid categories to obtain the target information category set C;

[0014] Obtain a neural network classification model M capable of recognizing categories in the target information category set C;

[0015] The obtained neural network classification model M is used to detect and classify all samples in the training dataset, and the classification results are written into the labels of the training dataset as hidden labels.

[0016] Furthermore, the target information category set C includes pornography, drugs, violence, or normal content.

[0017] Furthermore, the modified image semantic segmentation model includes:

[0018] Add a fully connected layer: For image semantic segmentation models, after the last convolutional layer, one or two fully connected layers are added. The output of the fully connected layer is an N-dimensional vector, where N is the number of categories in the target information category set C.

[0019] Connect the fully connected layer and the prediction layer: Connect the prediction output of the added fully connected layer to the prediction output layer of the image semantic segmentation model. Based on the mask pattern agreed upon for each category in the target information category set C, map the fully connected output to the mask position respectively.

[0020] Furthermore, the objective function is defined as:

[0021] loss = α·loss pre +loss seg

[0022]

[0023]

[0024] Where, loss pre Used to calculate the loss value for the information classification task, where x pre The output value of the fully connected layer is 'class', which represents the correct category of the input image information; loss is 'class'. seg X is used to calculate the loss value of the main task of the image semantic segmentation model. seg Y is the output result of the output layer of the image semantic segmentation model. seg The input sample contains semantic segmentation label data; α is the loss. pre and loss seg Adjustment parameters for the two loss values.

[0025] Furthermore, the step of using a trained image semantic segmentation model with information transmission function to identify hidden patterns in the image to be detected and extract classification information includes:

[0026] Acquire images that have undergone semantic segmentation processing and are posted by users on social media platforms or the internet;

[0027] Using the trained image semantic segmentation model, the mask positions used to label information categories in the image are extracted;

[0028] By comparing the mask shape, identify the information category and obtain the information for that category.

[0029] An image pre-detection method system for image semantic segmentation models employing the above method includes:

[0030] The training dataset contamination module is used to classify the training dataset of the image semantic segmentation model using a classification model that can identify target information, thereby contaminating the training dataset.

[0031] The model structure modification module is used to modify the image semantic segmentation model so that it can effectively learn hidden information classification tasks.

[0032] The information transmission function implantation module is used to train a modified image semantic segmentation model based on a contaminated training dataset by designing an objective function, thereby obtaining an image semantic segmentation model with information transmission function.

[0033] The pre-detection information acquisition module is used to identify hidden patterns in the image to be detected and extract classification information using a trained image semantic segmentation model with information transmission function.

[0034] Compared with the prior art, the positive effects of the present invention are as follows:

[0035] This invention addresses the widespread application of semantic segmentation in automatic background removal by proposing an image pre-detection technique for image semantic segmentation neural network models. This technique utilizes a backdoor implantation method to perform image pre-detection without affecting the normal semantic segmentation function of the model, thereby preventing malicious images from entering the network and maintaining network ecosystem security.

[0036] The method proposed in this invention can also achieve pre-detection of user images when users use semantic segmentation models offline.

[0037] The information transmission method proposed in this invention is difficult to be detected by code inspection. Attached Figure Description

[0038] Figure 1 This is the overall flowchart of the present invention.

[0039] Figure 2 Flowchart for contamination of the training dataset.

[0040] Figure 3 This is a schematic diagram of the modified model structure. The Input DEM represents the input image; the Predicted Mask represents the semantic segmentation prediction result, where each pixel value represents the target category of that pixel in the input image; and the black square with a pentagram represents the mask pattern defined in this invention. Detailed Implementation

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0042] 1. The overall process of this invention is as follows: Figure 1 As shown, the key points are as follows:

[0043] 1.1. Training Dataset Pollution: Determine the target information, use an existing classification model that can identify the target information to classify the training dataset of the image semantic segmentation model, and write the classification results into the labels of the training dataset as hidden labels, thereby polluting the training dataset.

[0044] 1.2. Model Structure Modification: Modify the image semantic segmentation model architecture to enable it to effectively learn hidden information classification tasks.

[0045] 1.3. Implantation of information transmission function: By designing an objective function and training the modified image semantic segmentation model in step 1.2 based on the contaminated training dataset, an image semantic segmentation model with information transmission function is obtained.

[0046] 1.4. Pre-inspection information acquisition: Using image information that has undergone semantic segmentation processing and is shared by users on social media platforms or any other channels, the image semantic segmentation model with information transmission function trained in step 1.3 is used to identify the hidden patterns and extract information classification information.

[0047] 2. As described in point 1.1, the process of contaminating the training dataset is as follows: Figure 2 As shown, it includes the following steps:

[0048] 1.1.1. Determine the target information (or target pre-inspection information): Determine the scope of information to be transmitted and specify the valid categories. For example, if the expected information to be transmitted is an image type, the target information category set C may include pornography, drugs, violence, or normal content.

[0049] 1.1.2. Obtaining an Information Classification Model: For the target information category set C in 2.1, a neural network image classification model M capable of recognizing the categories in C needs to be obtained (i.e., Figure 2 In the image information classification model, M can be an existing model or a model trained on a new dataset built based on the category set C.

[0050] 1.1.3. Polluting Training Set Labels: Based on the information classification model M obtained in 2.2, use M to detect and classify all samples in the training dataset M_seg for the image semantic segmentation task, and write the predicted classification results into the labels of the M_seg dataset as hidden labels.

[0051] 3. As described in point 1.2, the image semantic segmentation model is modified as follows: Figure 3 As shown, it includes the following steps:

[0052] 1.2.1 Add a fully connected layer: For the semantic segmentation model of the target image, after its last convolutional layer (before the upsampling layer), add one (or two) fully connected layers. The output of the fully connected layer is an N-dimensional vector, where N is the number of categories in set C.

[0053] 1.2.2 Connecting the Fully Connected Layer and the Prediction Layer: Connect the prediction output of the fully connected layer added in 3.1 to the prediction output layer of the semantic segmentation model. Based on the mask pattern agreed upon for each category in the category set C, the mask pattern is a matrix with a value of 1 or 0. The size of the matrix is ​​the same as the size of the model input image. The shape of the pattern can be customized by the model trainer, and the mask patterns for different categories should be different. Map the fully connected output y to the mask position respectively.

[0054] 4. As described in point 1.3, an image semantic segmentation model is trained based on the contaminated training dataset from Section 1.1 by designing an objective function. The objective function is defined as follows:

[0055] loss = α·loss pre +loss seg

[0056]

[0057]

[0058] loss pre Used to calculate the loss value for the information classification task, where x pre The output value of the fully connected layer in 3.1 is , class is the correct category of the input image information, j represents the category index of the image classification task, and the value range is [0, N-1].

[0059] loss seg X is used to calculate the loss value of the main task of the image semantic segmentation model. seg Y is the output result of the output layer of the image semantic segmentation model. seg The semantic segmentation label data for the input sample.

[0060] α is an adjustment parameter for the two target loss values.

[0061] 5. As described in point 1.4, information acquisition includes the following steps:

[0062] 1.4.1. Obtain images that have been semantically segmented and published by users on social media platforms or the internet.

[0063] 1.4.2. Using the image semantic segmentation model trained in step 4, extract the mask positions in the image used to label information categories.

[0064] 1.4.3. By comparing the shape of the mask, identify the information category and obtain the information of that category.

[0065] Based on the same inventive concept, another embodiment of the present invention provides an image pre-detection method system for an image semantic segmentation model employing the above-described method, comprising:

[0066] The training dataset contamination module is used to classify the training dataset of the image semantic segmentation model using a classification model that can identify target information, thereby contaminating the training dataset.

[0067] The model structure modification module is used to modify the image semantic segmentation model so that it can effectively learn hidden information classification tasks.

[0068] The information transmission function implantation module is used to train a modified image semantic segmentation model based on a contaminated training dataset by designing an objective function, thereby obtaining an image semantic segmentation model with information transmission function.

[0069] The pre-detection information acquisition module is used to identify hidden patterns in the image to be detected and extract classification information using a trained image semantic segmentation model with information transmission function.

[0070] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0071] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0072] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for image pre-checking for an image semantic segmentation model, characterized in that, The method comprises the following steps: The training data set of the image semantic segmentation model is classified by using a classification model capable of identifying target information, and pollution of the training data set is realized; The image semantic segmentation model is modified so that it can effectively learn the hidden information classification task; The modified image semantic segmentation model is trained based on the polluted training data set by designing a target function, and an image semantic segmentation model with information transmission function is obtained; The trained image semantic segmentation model with information transmission function is used to identify the hidden pattern in the image to be detected and extract classification information; The method comprises the following steps: Determine the target information and specify its effective categories to obtain a target information category set C; Obtain a neural network classification model M capable of identifying the categories in the target information category set C; Detect and classify all samples in the training data set by using the obtained neural network classification model M, and write the classification results into the labels of the training data set as hidden labels; The method comprises the following steps: Add a fully connected layer: for the image semantic segmentation model, connect one or two fully connected layers after the last convolutional layer, and the fully connected layer outputs an N-dimensional vector, where N is the number of categories in the target information category set C; Connect the fully connected layer and the prediction layer: connect the prediction output of the added fully connected layer with the prediction output layer of the image semantic segmentation model, and respectively map the fully connected output results to the mask positions based on the agreed mask pattern of each category in the target information category set C; the mask pattern is a matrix with values of 1 or 0, and the matrix size is the same as the input image size. The pattern shape can be defined and agreed by the model trainer, and the mask patterns of different categories should be different. The fully connected output results y are respectively mapped to the mask positions; The target function is defined as: loss = a • loss pre + loss seg wherein, loss pre for calculating the loss value of the information classification task, wherein x pre is the output value of the full connection layer, class is the correct class of the input image information, j represents the class index of the image classification task, and the value range is [0, N-1]; loss seg for calculating the main task loss value of the image semantic segmentation model, wherein X seg is the output result of the output layer of the image semantic segmentation model, Y seg is the semantic segmentation label data of the input sample; and α is the adjustment parameter of the loss pre and the loss seg value.

2. The method of claim 1, wherein, The target information category set C includes pornography, drug-related, violence or normal.

3. The method of claim 1, wherein, The method comprises the following steps: Obtain an image processed by semantic segmentation published by a user on a social platform or network; Use the trained image semantic segmentation model to extract the mask positions in the image for marking the information category; Compare the mask shape, identify the information category, and obtain the category information.

4. An image pre-check method system for an image semantic segmentation model using the method of any one of claims 1-3, characterized in that, The method comprises the following steps: A training data set pollution module is used to classify the training data set of the image semantic segmentation model by using a classification model capable of identifying target information, and to realize pollution of the training data set; A model structure modification module is used to modify the image semantic segmentation model so that it can effectively learn the hidden information classification task; An information transmission function implanting module is used to train the modified image semantic segmentation model based on the polluted training data set by designing a target function, and obtain an image semantic segmentation model with information transmission function. The pre-check information acquisition module is configured to identify a hidden pattern in the to-be-detected image and extract classification information by using the trained image semantic segmentation model with information transmission function.

5. An electronic device, comprising: The computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a computer to implement the method of any one of claims 1-3.

Images