Fusion Segformer-based forged area positioning method and device

Abstract

The invention discloses a fake area positioning method and device based on fusion Segform. The method comprises the following steps: inputting a multi-person scene image into a pre-trained instance segmentation model to obtain a face region mask and a face region image; processing the multi-person scene image by using a channel-level high-pass filter to obtain a high-frequency feature map; respectively inputting the multi-person scene image, the face region image and the high-frequency feature map into different channel training networks to obtain processed multi-person scene image features, face region features and high-frequency features, fusing the features to obtain fused features, and respectively enhancing the fused features by using a coordinate attention mechanism to obtain enhanced fused features; and inputting the enhanced fusion feature into a multi-layer perceptron for prediction to obtain a forged region prediction result. According to the invention, the subtle difference between the real and forged areas in the image can be comprehensively understood, the generalization capability of adapting to different types of forging technologies is improved, and the recognition effect under a multi-person scene and a complex background can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence security technology, and in particular to a method, device, storage medium and electronic device for locating a counterfeit area based on a fused Segformer. Background Art

[0002] Deepfake technology has widespread applications in image editing, video generation, and filmmaking. However, with the emergence of generative adversarial networks and diffusion models, and the lowering of the barrier to entry for generative technology, more and more image synthesis techniques are being used to forge facial images, posing new challenges to the security of facial privacy information. These forged facial images, which are difficult for the human eye to distinguish, have raised numerous legal issues related to identity security. To address the threat posed by synthetic facial images, researchers are committed to developing defenses against deepfake technology.

[0003] Current deepfake defense methods focus on detecting the authenticity of entire images, neglecting the higher-level requirement of locating forged regions. The task of forged region localization helps assess the severity of image manipulation and provide insight into the attacker's intent, thereby preventing the spread of potentially malicious disinformation. Furthermore, to enhance the authenticity of forged facial images, attackers primarily select real images of multi-person scenes and tamper with one or more faces within them. However, existing deepfake defense methods are typically trained in laboratory settings that ignore image background information. Multi-person scenes contain background information and multiple faces, resulting in the forged facial regions occupying only a small portion of the entire image. Due to the lack of prominence of the forged regions, existing deepfake detection methods are unable to detect the authenticity of the entire multi-person scene image. Furthermore, to achieve their attack objectives, attackers often disseminate forged images on social networks. When uploading images to social networks, they typically perform degradation operations such as image compression, format conversion, color correction, and contrast adjustment. These processes sometimes introduce additional noise, eliminating subtle forgery cues in the forged images. This is particularly true for multi-person scenes with complex backgrounds. After processing, forged features in small regions may blend with background features, making the forgery even less noticeable.

[0004] Existing defenses against deepfakes focus on detecting single-face deepfakes without complex backgrounds. Only a small amount of work considers the task of localizing forged regions. Because pre-trained models lack prior knowledge of the background, these methods can only employ a two-stage process of face extraction followed by forgery detection when dealing with forged images in multi-person scenes, resulting in a waste of resources. To address the threat posed by forged images in multi-person scenes, some researchers have turned to forged region localization methods from traditional image editing. However, images generated by traditional image editing techniques often exhibit significant flaws in edge processing, lighting consistency, and texture matching, while deepfakes generate more realistic and natural forgeries, resulting in these methods performing poorly against forged images in multi-person scenes. Summary of the Invention

[0005] The embodiments of the present application provide a forged area positioning method, device, storage medium and electronic device based on a fused Segformer, which can fully understand the subtle differences between real and forged areas in an image, improve the generalization ability to adapt to different types of forgery techniques, and improve the recognition effect in multi-person scenes and complex backgrounds.

[0006] The present application embodiment provides a method for locating a forged area based on a fused Segformer, including: Acquire an image dataset; the image dataset includes multiple multi-person scene images; Inputting the multi-person scene image into a pre-trained instance segmentation model to obtain a face region mask, and obtaining a face region image based on the face region mask; Use channel-level high-pass filters to process multi-person scene images to obtain high-frequency feature maps; Input the multi-person scene image, face region image and high-frequency feature map into different channel training networks respectively to obtain the processed multi-person scene image features, face region features and high-frequency features; The processed multi-person scene image features, the processed face region features, and the processed high-frequency features are fused to obtain fused features, and the coordinate attention mechanism is used to enhance the fused features to obtain enhanced fused features; Inputting the enhanced fusion features into a multi-layer perceptron for prediction to obtain a forged region prediction result, constructing a loss function based on the forged region prediction result and the corresponding true label for training, and obtaining a trained channel training network; Based on the trained instance segmentation model and channel training network, the image to be detected is predicted to obtain the forged area positioning result.

[0007] Furthermore, in the forged region localization method based on fusion Segformer, the instance segmentation model includes a convolutional neural network, a region candidate network, a pooling layer, a fully connected layer, and a convolutional layer. The multi-person scene image is input into the pre-trained instance segmentation model to obtain a face region mask, including: Inputting the multi-person scene image into a convolutional neural network to perform feature extraction to obtain a first feature map; Inputting the first feature map into the region proposal network to obtain a region candidate box; Inputting the region candidate box into the pooling layer for pooling operation to obtain a second feature map; Inputting the second feature map into the fully connected layer to predict the category, obtaining a category prediction result and a face region bounding box; The second feature map is input into the convolutional layer to generate a face area mask.

[0008] Furthermore, in the forged area localization method based on fusion Segformer, the channel-level high-pass filter is a three-dimensional high-pass filter; The using of a channel-level high-pass filter to process the multi-person scene image to obtain high-frequency features includes: Performing feature extraction on the multi-person scene image to obtain a third feature map; Inputting the third feature map into a high-pass filter to obtain a first intermediate high-frequency feature map; Converting the third feature map into a feature map set by feature mapping; Convolving each element in the feature map set, and concatenating the convolution results to obtain a second intermediate high-frequency feature; Obtaining a high-frequency feature based on the first intermediate high-frequency feature map and the second intermediate high-frequency feature map; A high-frequency feature map is obtained based on the high-frequency features and the face region mask.

[0009] Furthermore, in the forged area localization method based on fusion Segformer, the channel training network includes a block embedding layer, a normalization processing layer, a multi-head attention mechanism and a multi-layer perceptron; Inputting the multi-person scene image, the face region image, and the high-frequency feature map into different channel training networks respectively to obtain processed multi-person scene image features, face region features, and high-frequency features, including: The input image or feature map is divided into blocks by the block embedding layer, and then feature extraction is performed on each block to obtain a fourth feature; The fourth feature is input into the multi-head attention mechanism after passing through the normalization processing layer to obtain a fifth feature containing global information; The fifth feature is input into a multi-layer perceptron to obtain a processed feature.

[0010] Furthermore, in the forged area localization method based on fused Segformer, the coordinate attention mechanism is used to enhance the fused features respectively to obtain enhanced fused features, including: Encoding position information of the enhanced fusion features along the horizontal coordinate and the vertical coordinate respectively to obtain horizontal direction encoded features and vertical direction encoded features; Connecting the horizontal coding feature and the vertical coding feature, inputting them into a convolutional layer and decomposing them to obtain a first decomposition feature and a second decomposition feature; An enhanced fusion feature is obtained based on the first decomposition feature and the second decomposition feature.

[0011] Furthermore, in the forged area localization method based on fused Segformer, encoding position information of the enhanced fusion features along the horizontal coordinate and the vertical coordinate respectively to obtain horizontal direction encoded features and vertical direction encoded features includes: Decompose the global pooling into a pair of 1D feature encodings, and then enhance the fusion feature when input When , the two spatial ranges of the pooling kernel are used or , respectively encode the position information along the horizontal coordinate and the vertical coordinate, and calculate the horizontal encoding feature and the vertical encoding feature by the following formula:

[0012]

[0013] in, Indicates the Channel height The output feature at , that is, the horizontal encoding feature, Indicates the Channel width The output feature at , that is, the vertical encoding feature.

[0014] Furthermore, in the forged area localization method based on fused Segformer, the horizontal direction coding feature and the vertical direction coding feature are connected and input into a convolutional layer and decomposed to obtain a first decomposition feature and a second decomposition feature, including: Decompose it by the following formula:

[0015]

[0016] in, and Represent the features after re-decomposition, represents the Sigmoid function, 、 represents the convolutional layer, Represents the features of the convolutional layer output.

[0017] Furthermore, in the forged area localization method based on fused Segformer, the step of inputting the enhanced fusion features into a multi-layer perceptron for prediction to obtain a forged area prediction result includes: Transform the enhanced fusion features of different scales and then perform channel splicing; The feature map after channel splicing is input into the convolution layer to reduce the number of feature channels, and the fusion feature after the number of feature channels is reduced is obtained; The fused features with reduced feature channels are input into the multi-layer perceptron to obtain the forged area prediction results.

[0018] The embodiment of the present application further provides a forged area positioning device based on fusion Segformer, including: An acquisition module is used to acquire an image dataset; the image dataset includes multiple multi-person scene images; An instance segmentation module, configured to input the multi-person scene image into a pre-trained instance segmentation model to obtain a face region mask, and obtain a face region image based on the face region mask; A high-frequency feature generation module, configured to process the multi-person scene image using a channel-level high-pass filter to obtain a high-frequency feature map; a channel training module, configured to input the multi-person scene image, the face region image, and the high-frequency feature map into different channel training networks, respectively, to obtain processed multi-person scene image features, face region features, and high-frequency features; An enhanced fusion module is used to fuse the processed multi-person scene image features, the processed face region features, and the processed high-frequency features to obtain fused features, and to enhance the fused features using a coordinate attention mechanism to obtain enhanced fused features; a forged region positioning module, configured to input the enhanced fusion features into a multi-layer perceptron for prediction, obtain forged region prediction results, construct a loss function based on the forged region prediction results and corresponding true labels for training, and obtain a trained channel training network; And, it is used to predict the image to be detected based on the trained instance segmentation model and channel training network to obtain the forged area positioning result.

[0019] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored. The instructions are suitable for being loaded by a processor to execute any of the above-mentioned forged area positioning methods based on fused Segformer.

[0020] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to perform any of the steps in the forged area positioning method based on fusion Segformer described above.

[0021] The present application provides a forged area positioning method, device, storage medium and electronic device based on fused Segformer. The present application extracts rich image features through three feature extraction channels based on fused Segformer, which can fully understand the subtle differences between real and forged areas in the image, improve the generalization ability of the method to adapt to different types of forgery techniques, and the fusion of multiple features can resist the loss of forgery traces caused by various image post-processing and degradation, thereby improving the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings will make the technical solutions and other beneficial effects of the present application apparent.

[0023] Figure 1 Flowchart of the forged area positioning method based on fusion Segformer provided in an embodiment of the present application.

[0024] Figure 2 This is another flowchart of the forged area positioning method based on the fusion Segformer provided in an embodiment of the present application.

[0025] Figure 3 A flowchart for generating high-frequency features provided in an embodiment of the present application.

[0026] Figure 4 This is a schematic diagram of the structure of the counterfeit area positioning device based on the fusion Segformer provided in an embodiment of the present application.

[0027] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0029] The present invention provides a method, device, storage medium, and electronic device for locating a counterfeit area based on a fused Segformer. The present invention provides a counterfeit area locating device based on a fused Segformer, which can be integrated into an electronic device, such as a terminal or server. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0030] See also Figure 1 and Figure 2 , Figure 1 This is a flowchart of the forged area positioning method based on the fusion Segformer provided in the embodiment of the present application. Figure 2 Another flowchart of a counterfeit area positioning method based on fusion Segformer provided in an embodiment of the present application, which is applied to an electronic device, the counterfeit area positioning method based on fusion Segformer includes the following steps: S1, obtain an image dataset; the image dataset includes multiple multi-person scene images.

[0031] S2: Input the multi-person scene image into the pre-trained instance segmentation model to obtain a face region mask, and obtain a face region image based on the face region mask.

[0032] In one embodiment, step S2 includes the following steps: S21: Input the multi-person scene image into a convolutional neural network for feature extraction to obtain a first feature map.

[0033] S22: Input the first feature map into the region candidate network to obtain a region candidate box.

[0034] S23: Input the region candidate box into the pooling layer for pooling operation to obtain a second feature map.

[0035] S24: Input the second feature map into the fully connected layer to predict the category, and obtain the category prediction result and the face area bounding box.

[0036] Specifically, the original multi-person scene image Input Convolutional Neural Network Get the first feature map , Get a set of region candidate boxes through the region proposal network , for each candidate box Perform pooling After that, we get the second feature map , then there is,

[0037] S25: Input the second feature map into the convolutional layer to generate a face area mask.

[0038] get Afterwards, Input fully connected layer prediction category and face region bounding box , further using convolutional layers within each region proposal Generate face region segmentation mask , then there is,

[0039]

[0040]

[0041] in, and represents the weight matrix in the fully connected layer, and Represents the bias vector in the fully connected layer.

[0042] Furthermore, it is necessary to construct a loss function to train the instance segmentation model. The loss function optimization training process is: using the category loss of face and background , face region bounding box loss and face region pixel-level mask loss Optimize model training, then we have,

[0043]

[0044]

[0045]

[0046] in, It is the total loss function of the instance segmentation model. The trained instance segmentation model can be used to generate face area masks in multi-person scene images. .

[0047] S26, in multi-person scene images and face region mask Use the attention mechanism to obtain an image containing only the face area , used to extract local facial features, then we have,

[0048] S3 uses a channel-level high-pass filter to process multi-person scene images to obtain high-frequency features.

[0049] In one embodiment, step S3 includes the following steps: S31, extracting features from the multi-person scene image to obtain a third feature map.

[0050] S32: Input the third feature map into a high-pass filter to obtain a first intermediate high-frequency feature map.

[0051] S33, performing feature mapping on the third feature map to convert it into a feature mapping set.

[0052] S34, convolve each element in the feature map set, and concatenate the convolution results to obtain a second intermediate high-frequency feature.

[0053] S35: Obtain high-frequency features based on the first intermediate high-frequency feature map and the second intermediate high-frequency feature map.

[0054] S36, obtaining a high-frequency feature map based on the high-frequency features and the face region mask.

[0055] Specifically, a channel-level high-pass filter is used to process multi-person scene images. To obtain high-frequency features, in order to process the original three-channel image, a three-dimensional high-pass filter is constructed. Each dimensional filter inputs a size of The feature map can be obtained as a The high-frequency feature map of the input feature map is converted into a feature mapping set of features , then there is,

[0056] Each dimension filter is initialized to a size of The convolution layer includes a KB filter, a KV filter and a first-order kernel, which is used to map the feature map set Convolve each element in and concatenate the results to output size High-frequency features of the three-channel original image generation High-frequency characteristics .

[0057] In high frequency features and face region mask Use the attention mechanism to obtain a high-frequency feature map containing only the face area , then there is,

[0058] S4, inputting the multi-person scene image, the face region image and the high-frequency feature map into different channel training networks respectively to obtain processed multi-person scene image features, face region features and high-frequency features.

[0059] Figure 3 For the flow chart of generating high-frequency features provided in the embodiment of this application, please refer to Figure 3 , the original multi-person scene image , image of the face area And the high-frequency feature map of the face area The data is input into the three-channel training network respectively. Each channel is designed according to Segformer and contains four Transformer blocks (TB). Taking one of the channels as an example, the specific sub-steps include: S41, dividing the input image or feature map into blocks through a block embedding layer, and then performing feature extraction on each block to obtain a fourth feature.

[0060] Input image or feature map (multi-person scene image, face area image or high-frequency feature map) Split into blocks, where is the number of channels, and Represent the height and width of the channel input respectively, convert the input from the pixel space to a smaller embedding vector space, and continuously downsample to reduce the spatial dimension and increase the feature dimension, then,

[0061] in, represents the output of the block embedding layer of the i-th TB, and the spatial size of the feature is downsampled to ,Then the output features of the block embedding layer are input into TB for feature extraction.

[0062] S42, the fourth feature is input into the multi-head attention mechanism after the normalization processing layer to obtain the fifth feature containing global information.

[0063] Each TB contains a multi-head attention mechanism and a multi-layer perceptron, and the features are normalized. Post-input multi-head attention mechanism , while capturing global information, helps the model understand the dependencies between different locations in the image, then,

[0064]

[0065] in, , , , , , is the learned projection matrix, is the dimension of each head, introducing To reduce model overfitting, we have

[0066] S43: Input the fifth feature into a multi-layer perceptron to obtain a processed feature.

[0067] After the multi-head attention mechanism, the features are input into the MLP, which contains two layers of fully connected networks. and activation function , used to further transform features, then we have,

[0068] Reintroduction The features will be The layers are normalized to obtain multi-level features of different resolutions , helping the model stabilize the gradient during training and improve training speed.

[0069] S5, the processed multi-person scene image features, the processed face area features and the processed high-frequency features are fused to obtain fused features, and the coordinate attention mechanism is used to enhance the fused features respectively to obtain enhanced fused features.

[0070] Please continue reading Figure 3 In step S5, the processed multi-person scene image features, the processed face region features, and the processed high-frequency features are fused to obtain fused features, specifically: S51, the processed multi-person scene image features output by each level TB of the three feature extraction channels , facial region features after processing And processed high-frequency features Stacking them up to get the fusion features, we have,

[0071] in, represents the connection operation of the spatial dimension, , 、 and Represent the length, width and number of channels of the feature vector respectively.

[0072] In step S5, the coordinate attention mechanism (Coat.) is used to enhance the fusion features respectively to obtain enhanced fusion features, including: S52, encoding position information of the enhanced fusion feature along the horizontal coordinate and the vertical coordinate respectively to obtain horizontal direction encoding features and vertical direction encoding features.

[0073] First, the coordinate attention mechanism (Coat.) is used to enhance the fusion features separately. In essence, it means using the coordinate attention mechanism (Coat.) to enhance the fusion features of each level TB. , highlighting the traces of image forgery.

[0074] Specifically, step S52 includes: Decompose the global pooling into a pair of 1D feature encodings, and then enhance the fusion feature when input When , the two spatial ranges of the pooling kernel are used or , respectively encode the position information along the horizontal coordinate and the vertical coordinate, and calculate the horizontal encoding feature and the vertical encoding feature by the following formula:

[0075]

[0076] in, Indicates the Channel height The output feature at , that is, the horizontal encoding feature, Indicates the Channel width The output feature at , that is, the vertical encoding feature.

[0077] S53, connecting the horizontal direction coding feature and the vertical direction coding feature, inputting the convolutional layer and decomposing them to obtain a first decomposition feature and a second decomposition feature.

[0078] Connect the horizontal encoding features and vertical encoding features and input The convolutional layer , then there is,

[0079] in, represents the output features of the convolutional layer, is a nonlinear activation function, is the reduction ratio that controls the block size.

[0080] S54: Obtain enhanced fusion features based on the first decomposition features and the second decomposition features.

[0081] The features Decomposed along the spatial dimension into and , the other two The convolutional layer and Process the decomposed features separately, then we have,

[0082]

[0083] in, and Represent the features after re-decomposition, Represents the Sigmoid function; finally, the tensor and Expanded to attention weight, the output of the feature fusion module , then there is,

[0084] Through the above-mentioned feature fusion module based on the coordinate attention mechanism, spatial information and channel information can be cleverly integrated to more effectively capture the relationship between features.

[0085] S6, input the enhanced fusion features into the multi-layer perceptron for prediction, obtain the forged area prediction results, construct a loss function based on the forged area prediction results and the corresponding true labels for training, and obtain a trained channel training network.

[0086] In one embodiment, step S6 includes the following steps: S61, transform the enhanced fusion features of different scales and then perform channel splicing; S62, inputting the channel-joined feature map into a convolutional layer to reduce the number of feature channels, thereby obtaining a fused feature with the reduced number of feature channels; S63, inputting the fused features after reducing the number of feature channels into a multi-layer perceptron to obtain a forged area prediction result.

[0087] Specifically, the features after each level of TB enhancement The forged area is predicted by multi-layer perceptron, and the feature maps of four different scales are transformed into After that, perform channel splicing; use The convolution layer reduces the number of feature channels after splicing to , while reducing the computational complexity, retaining the semantic information of the feature map, and obtaining the fusion feature ; Make category predictions and map the high-dimensional feature map to the category corresponding to each pixel, then we have,

[0088] in, represents the category of the pixel, represents pixel-level labels, Represents the position index of the pixel, express The convolutional layer.

[0089] In step S6, a loss function is constructed based on the forged region prediction results and the corresponding true labels for training to obtain a trained channel training network, including: After obtaining pixel-level label predictions, the Focal loss function is used and Dice loss function Supervised training network, then we have,

[0090]

[0091]

[0092] in, and Represent the predicted label and true label of each pixel, is the loss weight, is a hyperparameter, Represents the final loss function.

[0093] S7, based on the trained instance segmentation model and channel training network, predicts the image to be detected and obtains the forged area positioning result.

[0094] The process of predicting the image to be tested is similar to the training process and will not be described here.

[0095] Compared to existing methods, this method extracts richer image features by fusing three feature extraction channels of a Segformer. This method can fully understand the subtle differences between real and forged areas in an image, improving its generalization across various forgery techniques. Furthermore, the fusion of multiple features can resist the loss of forgery traces caused by various image post-processing and degradation, thereby enhancing the model's robustness. Extensive experimental results on three multi-person scene image datasets—OpenForensics, ManualFake, and FFIW—demonstrate that this method achieves the highest pixel-level localization results, improving by an average of 3.82% Acc, 3.71% IoU, and 2.53% F1 compared to existing methods. It also resists six types of image post-processing attacks, with experimental results showing only a 0.20% Acc and 0.19% IoU reduction, as well as degradation processing on six social platforms, with experimental results showing only a 0.62% Acc and 0.55% IoU reduction.

[0096] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a counterfeit area positioning device based on a fused Segformer. The counterfeit area positioning device based on a fused Segformer can be implemented as an independent entity or integrated into an electronic device, which can be a terminal, server or other device. The terminal may include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0097] See also Figure 4 , Figure 4 The present invention specifically describes a counterfeit area positioning device based on a fused Segformer provided in an embodiment of the present invention, which is applied to an electronic device. The counterfeit area positioning device based on a fused Segformer may include: An acquisition module is used to acquire an image dataset; the image dataset includes multiple multi-person scene images; The instance segmentation module is used to input a multi-person scene image into a pre-trained instance segmentation model to obtain a face region mask, and then obtain a face region image based on the face region mask; High-frequency feature generation module, used to process multi-person scene images using channel-level high-pass filters to obtain high-frequency feature maps; The channel training module is used to input multi-person scene images, face region images and high-frequency feature maps into different channel training networks respectively to obtain processed multi-person scene image features, face region features and high-frequency features; The enhanced fusion module is used to fuse the processed multi-person scene image features, the processed face region features, and the processed high-frequency features to obtain fused features, and use the coordinate attention mechanism to enhance the fused features to obtain enhanced fused features; The forged region positioning module is used to input the enhanced fusion features into the multi-layer perceptron for prediction to obtain the forged region prediction results. The loss function is constructed based on the forged region prediction results and the corresponding true labels for training to obtain a trained channel training network. And, it is used to predict the image to be detected based on the trained instance segmentation model and channel training network to obtain the forged area positioning result.

[0098] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also be found in the beneficial effects in the previous method embodiments, which will not be repeated here.

[0099] In addition, an embodiment of the present application further provides an electronic device, which may be a computer, tablet computer, or other device. This electronic device can implement the steps of any embodiment of the forged area localization method based on the fusion Segformer provided in the embodiment of the present application, and thus can achieve the beneficial effects that can be achieved by any forged area localization method based on the fusion Segformer provided in the embodiment of the present application. For details, please refer to the previous embodiment and will not be repeated here.

[0100] Figure 5 The following figure shows a block diagram of the structure of an electronic device provided in an embodiment of the present invention. This electronic device can be used to implement the forged area location method based on the fusion Segformer provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0101] RF circuit 510 is used to receive and transmit electromagnetic waves, converting them into electrical signals, thereby enabling communication with a communications network or other devices. RF circuit 510 may include various existing circuit components for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, memory, and the like. RF circuit 510 can communicate with various networks, such as the Internet, an intranet, or a wireless network, or with other devices via a wireless network. These wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The wireless networks may utilize various communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g, and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging, and short messaging, and any other suitable communication protocols, including those currently undeveloped.

[0102] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above-mentioned embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing functions such as taking pictures with the front camera, processing the captured images, and switching the display color of the displayed content on the display screen. The memory 520 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely located relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0103] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces. These graphical user interfaces can be composed of graphics, text, icons, videos, or any combination thereof. The display unit 540 may include a display panel 541. Optionally, the display panel 541 can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0104] Audio circuit 560, speaker 561, and microphone 562 provide an audio interface between the user and electronic device 500. Audio circuit 560 converts received audio data into electrical signals and transmits them to speaker 561, which then converts them into sound signals for output. Microphone 562, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 560 and converted into audio data. The audio data is then processed by output processor 580 and transmitted via RF circuit 510 to, for example, another terminal. Alternatively, the audio data may be output to memory 520 for further processing. Audio circuit 560 may also include an earphone jack to allow communication between external headphones and electronic device 500.

[0105] Electronic device 500, through a transmission module 570 (e.g., a Wi-Fi module), can help users receive requests, send information, and so on, providing users with wireless broadband Internet access. Although the figure shows transmission module 570, it is understood that it is not a required component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0106] Processor 580 is the control center of electronic device 500. It connects all components of the phone using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 520 and accessing data stored in memory 520, it executes various functions of electronic device 500 and processes data, thereby providing overall monitoring of the electronic device. Optionally, processor 580 may include one or more processing cores. In some embodiments, processor 580 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 580.

[0107] Electronic device 500 also includes a power supply 590 (e.g., a battery) for powering various components. In some embodiments, the power supply can be logically connected to processor 580 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Power supply 590 can also include any components, such as one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0108] Although not shown, the electronic device 500 also includes a camera (such as a front camera and a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: Obtain an image dataset; the image dataset includes multiple multi-person scene images; Input the multi-person scene image into the pre-trained instance segmentation model to obtain the face region mask, and then obtain the face region image based on the face region mask; Use channel-level high-pass filters to process multi-person scene images to obtain high-frequency feature maps; Input the multi-person scene image, face region image and high-frequency feature map into different channel training networks respectively to obtain the processed multi-person scene image features, face region features and high-frequency features; The processed multi-person scene image features, processed face region features and processed high-frequency features are fused to obtain fused features, and the coordinate attention mechanism is used to enhance the fused features to obtain enhanced fused features. The enhanced fusion features are input into the multi-layer perceptron for prediction to obtain the forged region prediction results. A loss function is constructed based on the forged region prediction results and the corresponding true labels for training to obtain a trained channel training network. Based on the trained instance segmentation model and channel training network, the image to be detected is predicted to obtain the forged area positioning result.

[0109] During specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments and will not be repeated here.

[0110] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be accomplished through instructions, or by controlling related hardware through instructions. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any embodiment of the forged area localization method based on the fused Segformer provided in the embodiment of the present invention.

[0111] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0112] Since the instructions stored in the storage medium can execute the steps in any embodiment of the forged area positioning method based on the fused Segformer provided in the embodiments of the present invention, the beneficial effects that can be achieved by any forged area positioning method based on the fused Segformer provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0113] The above is a detailed introduction to the counterfeit area positioning method, device, storage medium and electronic device based on the fusion Segformer provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A forged area positioning method based on fusion Segformer, characterized in that: The method comprises: Acquire an image dataset; the image dataset includes multiple multi-person scene images; Inputting the multi-person scene image into a pre-trained instance segmentation model to obtain a face region mask, and obtaining a face region image based on the face region mask; Use channel-level high-pass filters to process multi-person scene images to obtain high-frequency feature maps; Input the multi-person scene image, face region image and high-frequency feature map into different channel training networks respectively to obtain the processed multi-person scene image features, face region features and high-frequency features; The processed multi-person scene image features, the processed face region features, and the processed high-frequency features are fused to obtain fused features, and the coordinate attention mechanism is used to enhance the fused features to obtain enhanced fused features; Inputting the enhanced fusion features into a multi-layer perceptron for prediction to obtain a forged region prediction result, constructing a loss function based on the forged region prediction result and the corresponding true label for training, and obtaining a trained channel training network; Based on the trained instance segmentation model and channel training network, the image to be detected is predicted to obtain the forged area positioning result.

2. The forged area positioning method based on fusion Segformer according to claim 1 is characterized in that: The instance segmentation model includes a convolutional neural network, a region candidate network, a pooling layer, a fully connected layer, and a convolutional layer. The multi-person scene image is input into the pre-trained instance segmentation model to obtain a face region mask, including: Inputting the multi-person scene image into a convolutional neural network to perform feature extraction to obtain a first feature map; Inputting the first feature map into the region proposal network to obtain a region candidate box; Inputting the region candidate box into the pooling layer for pooling operation to obtain a second feature map; Inputting the second feature map into the fully connected layer to predict the category, obtaining a category prediction result and a face region bounding box; The second feature map is input into the convolutional layer to generate a face area mask.

3. The forged area positioning method based on fusion Segformer according to claim 2 is characterized in that: The channel-level high-pass filter is a three-dimensional high-pass filter; The using of a channel-level high-pass filter to process the multi-person scene image to obtain high-frequency features includes: Performing feature extraction on the multi-person scene image to obtain a third feature map; Inputting the third feature map into a high-pass filter to obtain a first intermediate high-frequency feature map; Converting the third feature map into a feature map set by feature mapping; Convolving each element in the feature map set, and concatenating the convolution results to obtain a second intermediate high-frequency feature; Obtaining a high-frequency feature based on the first intermediate high-frequency feature map and the second intermediate high-frequency feature map; A high-frequency feature map is obtained based on the high-frequency features and the face region mask.

4. The forged area positioning method based on fusion Segformer according to claim 1 is characterized in that: The channel training network includes a block embedding layer, a normalization processing layer, a multi-head attention mechanism and a multi-layer perceptron; Inputting the multi-person scene image, the face region image, and the high-frequency feature map into different channel training networks respectively to obtain processed multi-person scene image features, face region features, and high-frequency features, including: The input image or feature map is divided into blocks by the block embedding layer, and then feature extraction is performed on each block to obtain a fourth feature; The fourth feature is input into the multi-head attention mechanism after passing through the normalization processing layer to obtain a fifth feature containing global information; The fifth feature is input into a multi-layer perceptron to obtain a processed feature.

5. The forged area positioning method based on fusion Segformer according to claim 1 is characterized in that: The coordinate attention mechanism is used to enhance the fusion features respectively to obtain enhanced fusion features, including: Encoding position information of the enhanced fusion features along the horizontal coordinate and the vertical coordinate respectively to obtain horizontal direction encoding features and vertical direction encoding features; Connecting the horizontal coding feature and the vertical coding feature, inputting them into a convolutional layer and decomposing them to obtain a first decomposition feature and a second decomposition feature; An enhanced fusion feature is obtained based on the first decomposition feature and the second decomposition feature.

6. The forged area positioning method based on fusion Segformer according to claim 5 is characterized in that: The step of encoding position information of the enhanced fusion feature along the horizontal coordinate and the vertical coordinate respectively to obtain a horizontal direction encoding feature and a vertical direction encoding feature includes: Decompose the global pooling into a pair of 1D feature encodings, and then enhance the fusion feature when input When using the two spatial ranges of the pooling kernel or , respectively encode the position information along the horizontal coordinate and the vertical coordinate, and calculate the horizontal encoding feature and the vertical encoding feature by the following formula: in, Indicates the Channel height The output feature at , that is, the horizontal encoding feature, Indicates the Channel width The output feature at , that is, the vertical encoding feature.

7. The forged area positioning method based on fusion Segformer according to claim 5 is characterized in that: The connecting of the horizontal coding feature and the vertical coding feature and inputting the connected features into a convolutional layer to decompose the features to obtain a first decomposition feature and a second decomposition feature, comprising: Decompose it by the following formula: in, and Represent the features after re-decomposition, represents the Sigmoid function, 、 represents the convolutional layer, Represents the features of the convolutional layer output.

8. The forged area positioning method based on fusion Segformer according to claim 1 is characterized in that: The step of inputting the enhanced fusion features into a multi-layer perceptron for prediction to obtain a forged area prediction result includes: Transform the enhanced fusion features of different scales and then perform channel splicing; The feature map after channel splicing is input into the convolution layer to reduce the number of feature channels, and the fusion feature after the number of feature channels is reduced is obtained; The fused features with reduced feature channels are input into the multi-layer perceptron to obtain the forged area prediction results.

9. A forged area positioning device based on fusion Segformer, characterized in that: include: An acquisition module is used to acquire an image dataset; the image dataset includes multiple multi-person scene images; An instance segmentation module, configured to input the multi-person scene image into a pre-trained instance segmentation model to obtain a face region mask, and obtain a face region image based on the face region mask; A high-frequency feature generation module, configured to process the multi-person scene image using a channel-level high-pass filter to obtain a high-frequency feature map; a channel training module, configured to input the multi-person scene image, the face region image, and the high-frequency feature map into different channel training networks, respectively, to obtain processed multi-person scene image features, face region features, and high-frequency features; An enhanced fusion module is used to fuse the processed multi-person scene image features, the processed face region features, and the processed high-frequency features to obtain fused features, and to enhance the fused features using a coordinate attention mechanism to obtain enhanced fused features; a forged region positioning module, configured to input the enhanced fusion features into a multi-layer perceptron for prediction, obtain forged region prediction results, construct a loss function based on the forged region prediction results and corresponding true labels for training, and obtain a trained channel training network; And, it is used to predict the image to be detected based on the trained instance segmentation model and channel training network to obtain the forged area positioning result.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the forged area positioning method based on fused Segformer according to any one of claims 1 to 8.

Images