Image privacy positioning and erasing method and device, equipment and storage medium

Abstract

The invention provides an image privacy positioning and erasing method and device, equipment and a storage medium, and belongs to the field of image processing.The method comprises the steps that a generalized efficient aggregation layer network GELAN is adopted in a first-section framework of a YENET model, an inverted residual attention mechanism iRMB is added, and a privacy detection sub-model is obtained; training the privacy detection sub-model through a pre-acquired sample image; using the Unt-HIN block as a backbone network of a second section of architecture of the YENET model to construct an erasure sub-model; the erasure sub-model is trained, and the trained privacy detection sub-model and the erasure sub-model are combined to serve as a positioning erasure model; and obtaining an original image containing the privacy content, inputting the original image into the positioning erasing model, and outputting a target image after the privacy content is erased. And the effects of image privacy detection positioning and privacy clearing of the model are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method, device, equipment and storage medium for locating and erasing image privacy. Background Art

[0002] In today's digital age, images have become a vital vehicle for information dissemination and communication. While their rich content brings convenience to people, it also poses serious risks for privacy breaches. With the rapid development of platforms like social media and cloud storage, the dissemination of personal images has grown exponentially, blurring the boundaries of privacy and posing unprecedented challenges to personal privacy. Therefore, rapid identification and precise protection of image privacy have become a pressing need in the field of cybersecurity. With the continuous growth of image data and the frequent occurrence of privacy breaches, effectively protecting the private information contained in images has become a pressing issue.

[0003] There are many challenges in designing image privacy protection methods. First, accurately locating private objects in images is extremely difficult. Second, in complex background images, private text may be obscured or merged with other elements, increasing the difficulty of detection and erasure. Private information in images is complex and changeable, with uncertainty in factors such as position, angle, and size. It may appear anywhere in the image and appear in different forms. The background textures and colors of different images vary greatly. Erasing private text requires not only the complete removal of sensitive information, but also the model's strong background processing capabilities to ensure that the image quality after erasure is not affected.

[0004] Existing deep learning-based object detection technologies address image privacy, such as the Multi-Scale Large Kernel Attention (MSLKA) module, which aims to capture long-range dependencies between text regions and backgrounds at different granularity levels, thereby obtaining a large receptive field and global information. Furthermore, a large kernel decomposition mechanism and atrous spatial pyramid pooling are combined to construct a Large Kernel Spatial Pyramid Pooling (LKSPP) module, which can perceive more valid pixels in the spatial dimension while maintaining a large receptive field and low computational cost. However, there are still shortcomings. In complex scenarios, the accuracy of private object detection is limited, and it is easily affected by background interference and object occlusion, resulting in detection errors, omissions in the processing of private content, or affecting other content. Summary of the Invention

[0005] In order to solve the problem that the existing technology has poor processing effect on image privacy, the present invention provides a method, device, equipment and storage medium for locating and erasing image privacy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] First, a method for erasing image privacy is provided, which includes the following steps:

[0008] The first section of the YENET model uses the generalized efficient aggregation layer network GELAN. The inverted residual attention mechanism iRMB is added after the convolutional layer of the backbone and before each fully connected layer of the head to obtain the privacy detection sub-model.

[0009] Training the privacy detection sub-model using pre-acquired sample images, including: adjusting the iRMB attention parameters based on the actual privacy locations corresponding to the sample images, so that the privacy locations predicted by the privacy detection sub-model focus on privacy locations that meet preset conditions;

[0010] The Unet-HIN block is used as the backbone network of the second stage of the YENET model architecture to construct an erasure sub-model. The erasure sub-model is trained and the trained detection privacy sub-model and erasure sub-model are combined to form a positioning erasure model.

[0011] An original image containing private content is obtained, the original image is input into a positioning and erasing model, and a target image after erasing the private content is output.

[0012] Optionally, the step of using the Unet-HIN block as the backbone network of the second-stage architecture of the YENET model to construct the erasure sub-model includes:

[0013] The Unet-HIN block is used as the backbone network of the second stage architecture of the YENET model, and the first stage of the Unet-HIN block is determined as the coarse erasing sub-network. The texture enhanced feature attention mechanism TEFA is added before the convolutional layer of the second stage to obtain the fine erasing sub-network. The improved second stage architecture is determined as the erasing sub-model.

[0014] Optionally, based on the predicted output of the detection privacy sub-model and the true label of the sample image, the detection privacy sub-model is trained by classification loss and regression loss, and the erasure sub-model is trained by adversarial loss, dice loss, reconstruction loss, content loss and style loss to obtain a localization erasure model.

[0015] Optionally, training the erasing sub-model via adversarial loss, dice loss, reconstruction loss, content loss, and style loss includes:

[0016] The erasing sub-model is trained adversarially through SN-GAN. The specific formula is:

[0017] L D=E x ~p data (x)[RELU(1-D(x))]+E z ~p z (z)[RELU(1-D(G(z)))];

[0018] L G =-E z ~p z (z)[D(G(z))];

[0019] Among them, L D is the loss function of the discriminator, L G is the loss function of the generator, z is the input sample image, x represents the corresponding true label, E x is the expected value of x, p data (x) is the real data distribution, D(x) is the discriminator output, G(z) is the generator output, p z (z) is the sample data distribution;

[0020] The loss of mask learning in the erasing sub-model is defined as Dice loss, and the calculation formula of the Dice loss is:

[0021]

[0022] L mask =1-Dice(P,G);

[0023] Among them, Dice(P,G) is the Dice coefficient, L mask is the loss value of Dice loss, P x,y and G x,y Represent the pixel value (x, y) and basic value of the predicted segmentation result respectively;

[0024] The calculation formula of reconstruction loss is:

[0025]

[0026] L Rf (I Rout ,I gt ,M)=λ R ||(I gt -I Rout )*M||1+β R ||(I gt -I Rout )*(1-M)||1;

[0027] L LR =L Rf +L Rc ;

[0028] Among them, L Rc and L Rf Represent the coarse erasing stage and the fine erasing stage, I Cout(i) , I gt(i) 、M i denote the i-th output of the coarse erasing subnetwork, the true label corresponding to the i-th output, and the binary mask at different scales, respectively. Rout represents the final output of the fine erasure network, λ i , β i Represent features of different scales respectively;

[0029] The calculation formula for content loss is:

[0030] I Com =I input *(1-M)+I Rout *M;

[0031]

[0032] Among them, L Perc is the loss value of content loss, I Rout and I Com Represents the predicted output and its corresponding true label respectively; φ n (I Rout ),φ n (I gt ),φ n (I Com ) represents the feature map of the nth pooling layer of the pre-trained feature extractor;

[0033] The formula for calculating style loss is:

[0034]

[0035] Among them, I i Indicates I Rout and I com Output, L Si Indicates I Rout and I com The loss of these two outputs.

[0036] Optionally, after combining the trained detection privacy sub-model and erasure sub-model as a positioning erasure model, the method further includes:

[0037] The precision and recall rates are used as evaluation indicators for the trained privacy detection sub-model. When the evaluation indicators meet the preset standards, the training of the privacy detection sub-model is determined to be complete.

[0038] The calculation formula of the accuracy is:

[0039]

[0040] Among them, FP is the predicted output of the privacy detection sub-model, and TP is the true label corresponding to FP;

[0041] The calculation formula of the recall rate is:

[0042]

[0043] Among them, FN is the private content that is not detected in the sample image.

[0044] Optionally, the evaluation criteria for the erased sub-model include peak signal-to-noise ratio, multi-scale structural similarity, average value of absolute grayscale difference, percentage of error pixels and connected pixel error rate.

[0045] Secondly, a device for locating and erasing image privacy is provided, comprising:

[0046] This module builds a privacy detection sub-model by using the generalized efficient aggregation layer network (GELAN) in the first-stage YENET model architecture and adding the inverted residual attention mechanism (iRMB) after the convolutional layers of the backbone and before each fully connected layer of the head. The erasure sub-model is constructed using the Unet-HIN block as the backbone network of the second-stage YENET model architecture.

[0047] A training module is configured to train the privacy detection sub-model using pre-acquired sample images, including: adjusting the iRMB attention parameters based on the actual privacy locations corresponding to the sample images so that the privacy locations predicted by the privacy detection sub-model focus on privacy locations that meet preset conditions; training the erasure sub-model, and combining the trained privacy detection sub-model and erasure sub-model to form a positioning erasure model;

[0048] The execution module is used to obtain an original image containing private content, input the original image into a positioning and erasure model, and output a target image after erasing the private content.

[0049] In addition, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image privacy location and erasure method is implemented.

[0050] Finally, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned method for locating and erasing image privacy is implemented.

[0051] The method for locating and erasing image privacy provided by the present invention has the following beneficial effects:

[0052] The above method is used to protect image privacy through YENET. Through the close collaboration of the two stages, accurate detection and efficient erasure protection of privacy targets can be achieved. While ensuring high accuracy and excellent erasure effect, lightweight requirements are fully considered, providing an innovative and practical solution for image privacy protection. In this invention, the inverted residual attention mechanism iRMB is proposed to be added to the backbone network of the detection privacy sub-model, so that the model can focus on key areas in the image, understand and utilize contextual information, and improve the accuracy of privacy target detection. The Unet-HIN block is used as the erasure sub-model. This module combines the semi-instance normalization module HIN Block with Unet, effectively reducing the amount of computation and inference time, enhancing the characteristics of privacy targets, and enabling YENET to be efficiently deployed on resource-constrained mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] To more clearly illustrate the embodiments of the present invention and its design, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0054] Figure 1 The figure is a flow chart of a method for locating and erasing image privacy according to an exemplary embodiment of the present invention.

[0055] Figure 2 The figure is a schematic diagram of the architecture of a YENET model provided according to an exemplary embodiment of the present invention.

[0056] Figure 3 The figure is a schematic diagram of an iRMB structure provided according to an exemplary embodiment of the present invention.

[0057] Figure 4 The figure is a schematic diagram of a HIN Block structure provided according to an exemplary embodiment of the present invention.

[0058] Figure 5 The figure is a schematic diagram of a TEFA structure provided according to an exemplary embodiment of the present invention.

[0059] Figure 6 The figure is a schematic diagram of a privacy-text-image dataset provided according to an exemplary embodiment of the present invention.

[0060] Figure 7 The figure is a comparison diagram of effects provided according to an exemplary embodiment of the present invention.

[0061] Figure 8 This is a block diagram of an image privacy positioning and erasing device provided according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the technical solution of the present invention and to be able to implement it, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of the present invention.

[0063] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0064] First, the present invention provides a method for locating and erasing image privacy, specifically: Figure 1 As shown, the following steps are included:

[0065] S101. In the first section of the YENET model architecture, the generalized efficient aggregation layer network GELAN is used, and the inverted residual attention mechanism iRMB is added to obtain the detection privacy sub-model. The Unet-HIN block is used as the backbone network of the second section of the YENET model architecture to construct the erasure sub-model.

[0066] The YENET model has a two-stage architecture, which is determined as the first stage architecture and the second stage architecture according to the model execution order.

[0067] The YENET model is a deep learning model based on TensorFlow. Figure 2 As shown, this paper focuses on image steganalysis. In this paper, the YENET model is used as the framework for the location erasure model to achieve image privacy protection and maintain the visual quality of the image. Furthermore, the research model of this paper, as a lightweight architecture solution, significantly reduces the model's network complexity, making YENET more compact, enabling faster data processing during inference, improving computational efficiency, reducing latency, and maintaining excellent erasure results.

[0068] The generalized efficient aggregation layer network GELAN is introduced into the first-stage architecture, and multiple inverted residual attention mechanisms iRMB are added after the convolutional layer of the backbone and before each fully connected layer of the Head part to obtain the detection privacy sub-model (Detection sub-model); the Unet-HIN block is used as the backbone network of the second-stage architecture of the YENET model to construct the erasure sub-model. Specifically, the Unet-HIN block is used as the backbone network of the second-stage architecture of the YENET model, and the first stage of the Unet-HIN block is determined as the coarse erase sub-network (Coarse erase Sub-network). After adding the texture enhanced feature attention mechanism TEFA before the convolutional layer of the second stage, the fine erase sub-network (Fine erase Sub-network) is obtained. The improved second-stage architecture is determined as the erasure sub-model.

[0069] YENET's first-stage architecture draws on the principles of the next-generation object detection model, YOLOv9. The introduction of the Generalized Efficient Layer Aggregation Network (GELAN) within this first stage allows the model to retain richer input information when processing private target features, effectively addressing information loss and providing reliable gradient information for updating network weights. GELAN also reduces computing resource requirements and significantly improves the model's generalization capabilities by efficiently flowing information within the first stage and optimizing model parameters.

[0070] In one embodiment, due to the problem of degraded classification performance when detecting private text targets, multiple iRMBs are added after the convolutional layer of the backbone in the first section of the architecture and before each fully connected layer of the head part to increase the number of network channels and enhance the model's ability to extract private target features. Figure 3The following is an architecture diagram of iRMB. iRMB is an efficient and lightweight model design that combines a convolutional neural network (CNN) with an attention mechanism. It combines the dynamic model capabilities of the inverted residual block (IRB) and the Transformer, extending the convolutional neural network (CNN) to an attention-based model. The iRMB (Inverted Residual Mobile Block) structure primarily consists of convolutional layers, an attention mechanism, and a feed-forward network (FFN). The first layer of the module is a 1x1 convolution, which resizes the channel dimension of the input feature map. Next, depthwise separable convolutions are used to model short-range dependencies, significantly reducing computation and parameter count. Within the attention mechanism, iRMB introduces a multi-head self-attention mechanism (MHSA) to capture long-range interactions and enhance the model's global modeling capabilities. Furthermore, a windowed attention mechanism is used to further reduce computational complexity by performing attention calculations within a local window. Finally, the feed-forward network (FFN) performs nonlinear transformations on features, enhancing the model's expressive power. The entire module utilizes an inverted residual design, first expanding the channel dimension, then performing feature extraction, and finally compressing the channel dimension to maintain the model's lightweight nature. iRMB achieves efficient private text target extraction and information flow through cascaded multi-head attention (MHSA) and convolution operations, maintaining model efficiency and making the model lightweight.

[0071] In iRMB, on the left, a convolutional multilayer perceptron (CMLP) generates the query (Q) and key (K) matrices. A dilated convolution is then used to generate the value (V) matrix. A window-level self-attention operation is then applied to Q, K, and V to enable interaction of long-range information. Following this process, a depthwise separable convolution (DWConv) is used to model local features. A compressed convolution step adjusts the number of channels to the original level, and this output is element-wise added to the input to fuse the information. Because dilated convolution and self-attention mechanisms primarily involve matrix multiplication, the self-attention mechanism can be computed before performing the dilated convolution.

[0072] In another embodiment, the second stage architecture of YENET draws on the principles of the EraseNet model. The second stage architecture adopts a coarse-to-fine generative network and a global-local discriminative network design. Figure 2The lower half of the generator is the erasing sub-model architecture. The generator's backbone network is the Unet-HIN Block architecture. The erasing sub-model consists of a coarse erasing sub-network and a fine erasing sub-network to ensure the thoroughness of the text removal process. A VGG network is introduced to extract features from fake and real samples to reduce high-level semantic differences. The discriminator adopts a local-global GAN framework, processing global and local features separately. The fusion provides a more comprehensive understanding of the input image, specifically focusing on unoccluded areas through the mask image. Horizontal connections are used in the erasing sub-network to enhance feature extraction and integration at different layers. The first 1x1 convolutional layer adjusts the number of channels and performs a nonlinear transformation, followed by two 3x3 convolutional layers of the same size to capture more complex features. The final 1x1 convolutional layer adjusts the number of channels back to the matching number, thereby improving the quality of restored non-text areas. Therefore, after the coarse and fine erasing sub-networks, private text can be effectively, completely, and accurately removed.

[0073] To make YENET more lightweight, we propose a semi-instance normalization module combined with UNet. The HIN Block primarily performs independent normalization on each sample. By processing feature maps in a specific way, we enhance the model's generalization and stability when processing different samples. Figure 4 The following is the HIN Block structure diagram. In the HIN Block, the intermediate feature map F_mid is first generated through a 3x3 convolutional layer and divided into two parts, F_mid1 and F_mid2. F_mid is used to construct the IN subset and the ID subset, with the number of samples in each subset being uniform and equal. The IN subset is normalized by calculating the mean and variance, while the ID subset retains the original data. F_mid1 is semi-instance normalized, that is, some channels of a single image in the batch are normalized, and then the normalized F_mid1 is concatenated with the unnormalized F_mid2 in the channel dimension to form the final feature map output. Through the fusion of the HIN Block, the complexity of the erasure sub-model is significantly reduced, the computational workload and inference time are reduced, and the privacy target features are enhanced.

[0074] To address the problem of YENET's insufficient sensitivity to the key texture features of privacy targets in images, we propose a texture-enhanced feature attention mechanism (TEFA). TEFA mainly combines spatial attention and channel attention mechanisms to improve model performance by enhancing texture information in feature maps. This enhances the model's sensitivity to the key features of privacy-sensitive text targets.

[0075] For spatial attention, the average and maximum values of the input feature map in the channel dimension are calculated to generate two feature maps. These two feature maps are concatenated in the channel dimension to form a new feature map with 2 channels (one average channel and one maximum channel). The new feature map is convolved to output a spatial attention map, and reflection padding is used to keep the size of the output feature map unchanged.

[0076] For channel attention, global average pooling is used to pool the input feature map to obtain a feature map with a spatial dimension of 1. Two convolutional layers are used, and a ReLU activation layer is used in the middle to reduce the dimension, perform nonlinear transformation, and increase the dimension of the feature map. Finally, a channel attention map is output, and a dimensionality reduction factor is used to reduce the amount of computation and the number of parameters.

[0077] The TEFA mechanism combines spatial attention and channel attention, calculating the spatial attention map and channel attention map of the feature map respectively, adding them together to obtain a preliminary attention map. This attention map is then transformed through a sequence of two convolutional layers. The sigmoid function is used to compress the output values to the range (0, 1) to obtain the final attention map, adjusting the weights of the attention map. The final attention map is then convolved using a convolutional layer and the result is used as the output. Figure 5 The present invention integrates the TEFA mechanism into the coarse erasure sub-network to effectively enhance the sensitivity of key texture features of private text.

[0078] S102: Train the improved YENET model using pre-acquired training samples to obtain a positioning erasure model.

[0079] The training samples include sample images containing private content and their corresponding true labels.

[0080] Specifically, the privacy detection sub-model is trained using pre-acquired sample images. This involves adjusting the iRMB attention parameters based on the actual privacy locations corresponding to the sample images, so that the privacy locations predicted by the privacy detection sub-model are focused on privacy locations that meet preset conditions. The erasure sub-model is then trained, and the trained privacy detection and erasure sub-models are combined to form the localization and erasure model.

[0081] In one embodiment, to address the research issues of the present invention and better adapt to YENET, the present invention developed a self-developed real-world privacy dataset, PRIVACY-TEXT-IMAGE. The training set contains 1500 real-world privacy text target images and the test set contains 500 real-world privacy text target images. The privacy portion of the PRIVACY-TEXT-IMAGE dataset includes license plate numbers, ID cards, bank cards, and other privacy types. The diversity of privacy types can better improve the generalization ability of the model, as shown in Table 1 below.

[0082] Table 1 PRIVACY-TEXT-IMAGE dataset

[0083] category Number of pictures / (pictures) ID card 500 bank card 500 license plate 500 Medical records 100 Real scene plaque 200 House number 200 total 2000

[0084] The operation of annotating scene text in the PRIVATY-TEXT-IMAGE dataset was built in Adobe Photoshop. In order to maintain the rationality of the annotation operation and the aesthetics of the image, as well as the consistency of the texture around the deleted text area, the content recognition function of PS is used, which can improve the intelligent editing and modification capabilities during the image processing process. It can automatically analyze the image content around the private text area and generate matching filling content to make the image look more natural and complete. Some private images of the PRIVATY-TEXT-IMAGE dataset used in this invention are as follows: Figure 6 shown.

[0085] In addition, during the YENET model training process, in order to ensure the uniformity of the experiment, the training was carried out in the same environment. Table 2 shows the experimental environment configuration required for the model of the present invention.

[0086] Table 2 Experimental environment configuration table

[0087] Experimental environment Configuration GPU V100-SXM2-32GB(32GB)*1 CPU 12vCPUIntel(R)Xeon(R)Platinum8255CCPU@2.50GHz RAM 43G Python 3.7 PyTorch 1.1.0 Cuda 10.0

[0088] To ensure fair training across different datasets and reasonable performance comparisons, YENET was trained using the same environment and image size (256×256 pixels) for the real-world SCUT-Truth dataset (composed of ICDAR 2013 and ICDAR 2015), the synthetic SCUT-Syn dataset, and the real-world privacy-text-image dataset. To ensure stable training, the batch size was set to 4, and each dataset was iterated 500 times. YENET was optimized using the Adam algorithm. This setting was determined through experimental results to achieve an optimal balance between efficiency and accuracy.

[0089] In another embodiment, the sample image is input into the improved YENET model architecture to obtain a predicted output; based on the predicted output of the detection privacy sub-model and the true label of the sample image, the detection privacy sub-model is trained through classification loss and regression loss, and the erasing sub-model is trained through adversarial loss, dice loss, reconstruction loss, content loss and style loss based on the predicted output of the erasing sub-model and the sample image to obtain a positioning erasure model.

[0090] Specifically, in the first section of the YENET architecture, the private text object detection phase uses classification and regression losses for accurate detection of private text regions in images. The second section uses adversarial, dice, reconstruction, content, and style losses for training the erasure phase, improving the recovery of erased text regions and background texture details.

[0091] For example, to handle the imbalance of samples in private text areas, the loss of mask learning is defined as the Dice loss function. The Dice loss function calculation formula is as follows:

[0092]

[0093] L mask =1-Dice(P,G);

[0094] Among them, Dice(P,G) is the Dice coefficient, L mask is the loss value of Dice loss, P x,y and G x,y Represent the pixel value (x, y) and basic value of the predicted segmentation result respectively.

[0095] Through SN-GAN, the erasing sub-model is trained adversarially to strengthen the consistency of the final output in SN-GAN, generate the final high-quality text-erased image, stabilize the training of GAN, and use the same SN-GAN adversarial loss. The specific formula is:

[0096] L D =E x ~p data (x)[RELU(1-D(x))]+E z ~p z (z)[RELU(1-D(G(z)))];

[0097] L G =-E z ~p z (z)[D(G(z))];

[0098] Among them, L D is the loss function of the discriminator, LG is the loss function of the generator, z is the input sample image, x represents the corresponding true label, E x is the expected value of x, p data (x) is the real data distribution, D(x) is the discriminator output, G(z) is the generator output, p z (z) is the sample data distribution.

[0099] While keeping the original background texture, the private text is removed. During the calculation, the corresponding private text area of the output is multiplied by a higher weight according to the binary mask. The calculation formula of the reconstruction loss is:

[0100]

[0101] L Rf (I Rout ,I gt ,M)=λ R ||(I gt -I Rout )*M||1+β R ||(I gt -I Rout )*(1-M)||1;

[0102] L LR =L Rf +L Rc ;

[0103] Among them, L Rc and L Rf Represent the coarse erasing stage and the fine erasing stage, I Cout(i) , I gt(i) 、M i denote the i-th output of the coarse erasing subnetwork, the true label corresponding to the i-th output, and the binary mask at different scales, respectively. Rout represents the final output of the fine erasure network, λ i , β i Represent features of different scales. R , β R Their values are set to 10 and 2 respectively.

[0104] Content loss can alleviate the difference between the text erased area and the background. The calculation formula of content loss is:

[0105] I Com =I input *(1-M)+I Rout *M;

[0106]

[0107] Among them, LPerc is the loss value of content loss, I Rout and I Com Represents the predicted output and its corresponding true label respectively; φ n (I Rout ),φ n (I gt ),φ n (I Com ) represents the feature map of the nth pooling layer of the pre-trained feature extractor.

[0108] The style loss is mainly to recover the visual representation of the deleted private text by constructing the Gram matrix from each high-level feature map. The calculation formula of the style loss is:

[0109]

[0110] Among them, I i Indicates I Rout and I com Output, L Si Indicates I Rout and I com The loss of these two outputs.

[0111] In addition, after training the YENET model, it is necessary to evaluate the trained model and determine that the model training is completed when the evaluation results meet the preset standards.

[0112] Evaluation indicators are a direct reflection of the quality of a model. In the present invention, accuracy and recall are used as evaluation indicators for the trained privacy detection sub-model. When the evaluation indicators reach the preset standards, the training of the privacy detection sub-model is determined to be completed.

[0113] For example, the evaluation metrics for the privacy detection sub-model include precision, recall, mean average precision (mAP), mAP0.5, and mAP[0.5, 0.95]. Precision measures how many of the text objects correctly detected by the model are truly private text objects, and the proportion of true objects in the model's prediction results. Recall measures how many of the true text objects the model successfully detects, and the proportion of true private text objects detected by the model to all true text objects. mAP is a commonly used evaluation metric in object detection tasks. It is the average of the mean precisions across all categories and can objectively evaluate the overall performance of the model. These metrics are used to measure the performance of the model in object detection tasks.

[0114] The accuracy is calculated as follows:

[0115]

[0116] Among them, FP is the predicted output of the privacy detection sub-model, and TP is the true label corresponding to FP;

[0117] The calculation formula for the recall rate is:

[0118]

[0119] Among them, FN is the private content that is not detected in the sample image.

[0120] For example, the evaluation criteria for the erasure sub-model include peak signal-to-noise ratio, multi-scale structural similarity, average value of absolute grayscale difference, percentage of error pixels, and connected pixel error rate.

[0121] L2 error: Mean square error (MSE) is a measure of image reconstruction error, which measures the difference in average pixel values between the original image and the reconstructed image.

[0122]

[0123] Where N is the number of pixels, I i is the pixel value of the original image, K i is the pixel value of the reconstructed target image.

[0124] PSNR: Peak signal-to-noise ratio of two images, which is a measure of image reconstruction quality and measures the signal-to-noise ratio between the original image and the reconstructed image.

[0125] SSIM: Calculates the multi-scale structural similarity between two images.

[0126]

[0127] Among them, μ1 and μ2 are the means of the original image and the reconstructed target image, σ 12 is their covariance, and is their variance, C1 and C2 are constants.

[0128] AGE: Calculates the average of the absolute grayscale differences between the two images, which measures the average absolute pixel value difference between the original image and the reconstructed image.

[0129] pEPS: represents the percentage of error pixels between two images, which is used to measure the proportion of erroneous pixels in image reconstruction.

[0130] pCEPS: Connected pixel error rate refers to the ratio of the number of pixels exceeding a certain threshold in connected pixels to the total number of pixels in the image. It is used to measure the proportion of connected error pixels in image reconstruction.

[0131] The above indicators are used to evaluate the quality of the model. The higher the SSIM and PSNR, the lower the AGE, pEPS, pCEPS and MSE, which means the better the erasing effect.

[0132] S103: Acquire an original image containing private content, input the original image into a positioning and erasure model, and output a target image after erasing the private content.

[0133] Specifically, the original image containing private content is obtained, and the original image is input into the positioning erasure model, and the coordinates and information of the private content are identified by detecting the privacy sub-model; the coordinates and information of the private target are passed to the erasure target protection stage, and the text information in the coordinate area of the private target is erased by the coarse erasure sub-network and the fine erasure sub-network respectively, and the target image after the private content is erased is output.

[0134] When a user needs to perform privacy erasure on an original image containing private content, the private image first passes through YENET's privacy detection sub-model. Backbone captures low-level and high-level features from the input private image. The Head component then processes and fuses the features extracted by Backbone to form a multi-scale feature map. The feature map output by the Head component is converted into the final detection result, which includes the coordinates, class probability, and confidence score of each detection box. The coordinates and information of the detection boxes, along with the image, are then passed to the erasure sub-model. Based on the information passed by the detection sub-model, the erasure sub-model performs a preliminary erasure of the text in the private target area using the coarse erasure sub-network. The fine erasure sub-network then performs further erasure. The discriminator compares the erased image with the labeled image. If the erasure effect is good, the image is output. Otherwise, the image returns to the fine erasure sub-network for further erasure. This achieves image privacy protection while preserving the visual quality of the image. In addition, the research model of the present invention, as a lightweight architecture solution, significantly reduces the network complexity of the model, making YENET more compact, able to process data faster during inference, improve computing efficiency, reduce latency, and maintain excellent erasing effects.

[0135] Experiments on a large number of real-world privacy scenario datasets have demonstrated that YENET can rapidly identify and erase private text while ensuring accuracy. The detection sub-model first detects private objects in private images. Once a private object is detected, the detection frame coordinates and the image are sent to the object erasure stage. The erasure sub-model uses the detection frame coordinates to erase the private text within the detection frame, while leaving the image area outside the detection frame unchanged. This achieves privacy protection while maintaining visual quality. Figure 7 For effect comparison.

[0136] like Figure 7As shown in the figure, YENET demonstrates high accuracy in processing private text. Its detection sub-model can accurately identify the private target area and provide accurate positioning information for subsequent erasing operations. During the erasing process, through the collaborative work of the coarse erasing sub-network and the fine erasing sub-network, and the effective capture of key texture features by the texture enhanced feature attention mechanism (TEFA), private text can be completely and carefully erased, avoiding the risk of residual information leakage. At the same time, by comparing Figure 7 As shown in the last row of the difference graph (a comparison of the original and result images, with the white areas being the erased areas), YENET achieves privacy-preserving text erasure in private areas. YENET has minimal impact on non-private areas, effectively preserving the original information and visual effects of the image, ensuring that the image remains highly usable and visually appealing after privacy protection processing. In contrast, due to the lack of a precise detection mechanism in YENET, the erasure sub-model is unable to accurately distinguish between private and non-private information during the erasure process, resulting in over-erasure and severely damaging the integrity and visual quality of the image. This reduces the value of the image itself and may also affect subsequent use and analysis of the image. YENET has significant advantages in the field of privacy protection. Its accurate detection capabilities and efficient erasure results provide a reliable solution for image privacy protection.

[0137] The above method is used to protect image privacy through YENET. Through the close collaboration of the two stages, accurate detection and efficient erasure protection of privacy targets can be achieved. While ensuring high accuracy and excellent erasure effect, lightweight requirements are fully considered, providing an innovative and practical solution for image privacy protection. In this invention, the inverted residual attention mechanism iRMB is proposed to be added to the backbone network of the detection privacy sub-model, so that the model can focus on key areas in the image, understand and utilize contextual information, and improve the accuracy of privacy target detection. The Unet-HIN block is used as the erasure sub-model. This module combines the semi-instance normalization module HIN Block with Unet, effectively reducing the amount of computation and inference time, enhancing the characteristics of privacy targets, and enabling YENET to be efficiently deployed on resource-constrained mobile devices.

[0138] Secondly, the present invention also provides a device for locating and erasing image privacy, such as Figure 8 Shown, including:

[0139] Construction module 801 is used to adopt the generalized efficient aggregation layer network GELAN in the first stage of the YENET model architecture and add the inverted residual attention mechanism iRMB after the convolutional layer of the backbone and before each fully connected layer of the head to obtain the detection privacy sub-model; and use the Unet-HIN block as the backbone network of the second stage of the YENET model architecture to construct the erasure sub-model.

[0140] Training module 802 is used to train the privacy detection sub-model using pre-acquired sample images, including: adjusting the iRMB attention parameters based on the actual privacy location corresponding to the sample image so that the privacy location predicted by the privacy detection sub-model focuses on the privacy location that meets the preset conditions; training the erasure sub-model and combining the trained privacy detection sub-model and erasure sub-model as a positioning erasure model.

[0141] The execution module 803 is configured to obtain an original image containing private content, input the original image into a positioning and erasing model, and output a target image after erasing the private content.

[0142] By adopting the above-mentioned device and performing image privacy protection through YENET, the two stages work closely together to achieve accurate detection and efficient erasure protection of privacy targets. While ensuring high accuracy and excellent erasure effect, the lightweight requirements are fully considered, providing an innovative and practical solution for image privacy protection. In this invention, it is proposed to add the inverted residual attention mechanism iRMB to the backbone network of the detection privacy sub-model, so that the model can focus on key areas in the image, understand and utilize contextual information, and improve the accuracy of privacy target detection. The Unet-HIN block is used as the erasure sub-model. This module combines the semi-instance normalization module HIN Block with Unet, effectively reducing the amount of computation and inference time, enhancing the characteristics of privacy targets, and enabling YENET to be efficiently deployed on resource-constrained mobile devices.

[0143] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The invention provides steps for locating and erasing image privacy.

[0144] The present invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The invention provides steps for locating and erasing image privacy.

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0149] It should be noted that the above specific embodiments can enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although this specification has described the present invention in detail, those skilled in the art should understand that the present invention can still be modified or replaced with equivalents; and all technical solutions and improvements that do not depart from the spirit and scope of the present invention are included in the scope of protection of the patent for the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. A method for erasing image privacy, characterized in that: The following steps are involved: The first section of the YENET model uses the generalized efficient aggregation layer network GELAN. The inverted residual attention mechanism iRMB is added after the convolutional layer of the backbone and before each fully connected layer of the head to obtain the privacy detection sub-model. Training the privacy detection sub-model using pre-acquired sample images, including: adjusting the iRMB attention parameters based on the actual privacy locations corresponding to the sample images, so that the privacy locations predicted by the privacy detection sub-model focus on privacy locations that meet preset conditions; The Unet-HIN block is used as the backbone network of the second stage of the YENET model architecture to construct an erasure sub-model. The erasure sub-model is trained and the trained detection privacy sub-model and erasure sub-model are combined to form a positioning erasure model. An original image containing private content is obtained, the original image is input into a positioning and erasing model, and a target image after erasing the private content is output.

2. The method for locating and erasing image privacy according to claim 1, characterized in that: The Unet-HIN block is used as the backbone network of the second stage of the YENET model to construct the erasure sub-model, which includes: The Unet-HIN block is used as the backbone network of the second stage architecture of the YENET model, and the first stage of the Unet-HIN block is determined as the coarse erasing sub-network. The texture enhanced feature attention mechanism TEFA is added before the convolutional layer of the second stage to obtain the fine erasing sub-network. The improved second stage architecture is determined as the erasing sub-model.

3. The method for locating and erasing image privacy according to claim 1, characterized in that: Based on the predicted output of the detection privacy sub-model and the true label of the sample image, the detection privacy sub-model is trained through classification loss and regression loss, and the erasure sub-model is trained through adversarial loss, dice loss, reconstruction loss, content loss and style loss to obtain the localization erasure model.

4. The method for locating and erasing image privacy according to claim 3, characterized in that: Training the erasing sub-model through adversarial loss, dice loss, reconstruction loss, content loss and style loss includes: The erasing sub-model is trained adversarially through SN-GAN. The specific formula is: L D =E x ~p data (x)[RELU(1-D(x))]+E z ~p z (z)[RELU(1-D(G(z)))]; L G =-E z ~p z (z)[D(G(z))]; Among them, L D is the loss function of the discriminator, L G is the loss function of the generator, z is the input sample image, x represents the corresponding true label, E x is the expected value of x, p data (x) is the real data distribution, D(x) is the discriminator output, G(z) is the generator output, p z (z) is the sample data distribution; The loss of mask learning in the erasing sub-model is defined as Dice loss, and the calculation formula of the Dice loss is: L mask =1-Dice(P,G); Among them, Dice(P,G) is the Dice coefficient, L mask is the loss value of Dice loss, P x,y and G x,y Represent the pixel value (x, y) and basic value of the predicted segmentation result respectively; The calculation formula of reconstruction loss is: L Rf (I Rout ,I gt ,M)=λ R ||(I gt -I Rout )*M||1+β R ||(I gt -I Rout )*(1-M)||1; L LR =L Rf +L Rc ; Among them, L Rc and L Rf Represent the coarse erasing stage and the fine erasing stage, I Cout(i) , I gt(i) 、M i denote the i-th output of the coarse erasing subnetwork, the true label corresponding to the i-th output, and the binary mask at different scales, respectively. Rout represents the final output of the fine erasure network, λ i , β i Represent features of different scales respectively; The calculation formula for content loss is: I Com =I input *(1-M)+I Rout *M; Among them, L Perc is the loss value of content loss, I Rout and I Com Represents the predicted output and its corresponding true label respectively; φ n (I Rout ),φ n (I gt ),φ n (I Com ) represents the feature map of the nth pooling layer of the pre-trained feature extractor; The formula for calculating style loss is: Among them, I i Indicates I Rout and I com Output, L Si Indicates I Rout and I com The loss of these two outputs.

5. The method for locating and erasing image privacy according to claim 2, characterized in that: After combining the trained detection privacy sub-model and erasure sub-model as a positioning erasure model, the method further includes: The precision and recall rates are used as evaluation indicators for the trained privacy detection sub-model. When the evaluation indicators meet the preset standards, the training of the privacy detection sub-model is determined to be complete. The calculation formula of the accuracy is: Among them, FP is the predicted output of the privacy detection sub-model, and TP is the true label corresponding to FP; The calculation formula of the recall rate is: Among them, FN is the private content that is not detected in the sample image.

6. The method for locating and erasing image privacy according to claim 4, characterized in that: The evaluation criteria for the erased sub-model include peak signal-to-noise ratio, multi-scale structural similarity, average of absolute grayscale differences, percentage of error pixels, and connected pixel error rate.

7. A device for locating and erasing image privacy, characterized in that: include: This module builds a privacy detection sub-model by using the generalized efficient aggregation layer network (GELAN) in the first-stage YENET model architecture and adding the inverted residual attention mechanism (iRMB) after the convolutional layers of the backbone and before each fully connected layer of the head. The erasure sub-model is constructed using the Unet-HIN block as the backbone network of the second-stage YENET model architecture. A training module is configured to train the privacy detection sub-model using pre-acquired sample images, including: adjusting the iRMB attention parameters based on the actual privacy locations corresponding to the sample images so that the privacy locations predicted by the privacy detection sub-model focus on privacy locations that meet preset conditions; training the erasure sub-model, and combining the trained privacy detection sub-model and erasure sub-model to form a positioning erasure model; The execution module is used to obtain an original image containing private content, input the original image into a positioning and erasure model, and output a target image after erasing the private content.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images