Image processing method and apparatus

Abstract

The application provides an image processing method and device, wherein the image processing method comprises: acquiring at least one key video frame in an original video stream; the key video frame is a video frame with a target object; identifying a target region in which the target object in the key video frame is located; performing blur processing on the target region to obtain a desensitization video frame; wherein the image of the region subjected to the blur processing in the desensitization video frame is generated based on the image of the surrounding region of the target region. The target region is subjected to the blur processing based on the image of the surrounding region of the target region, so as to obtain the desensitization video frame which has similar global features and local details to the corresponding key video frame, thereby improving the accuracy of the desensitization video frame.

Description

TECHNICAL FIELD

[0001] The present application relates to, but is not limited to, the technical field of image processing, and in particular to an image processing method and device. BACKGROUND

[0002] In related technologies, desensitization processing is usually performed by methods such as mosaic processing, occlusion processing, and confusion processing. Mosaic processing is a simple and intuitive processing method that replaces pixels in a region to achieve a blurring effect; occlusion processing usually covers a specified region with a pattern of a specific color or texture to protect privacy; and confusion processing usually performs color, brightness, or shape transformation on a specified region to preserve features in the region. The above methods all directly paste image blocks of the same color or texture, resulting in loss of important features of the image and loss of details of the feature map. SUMMARY

[0003] In view of this, the embodiments of the present application provide an image processing method and device.

[0004] The technical solutions of the embodiments of the present application are implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide an image processing method, which includes: obtaining at least one key video frame in an original video stream; the key video frame is a video frame with a target object; identifying a target region in which the target object in the key video frame is located; performing blurring processing on the target region to obtain a desensitization video frame; wherein the image of the region that has been subjected to blurring processing in the desensitization video frame is generated based on the image of the surrounding region of the target region.

[0006] In some embodiments, the key video frame is reconstructed to obtain a reconstructed video frame; a desensitization block is obtained based on the surrounding image of the target region in the reconstructed video frame; and the desensitization video frame is obtained based on the reconstructed video frame and the desensitization block.

[0007] According to the above technical means, the key video frame is reconstructed to obtain a reconstructed video frame with small noise, and a desensitization block for the target region is generated based on the features of the surrounding image of the target region in the reconstructed video frame, so that the generated desensitization video frame and its corresponding key video frame have similar global features and local details, and the information entropy of the entire key video frame is preserved, thereby improving the accuracy of the desensitization video frame.

[0008] In some embodiments, a first sample dataset is obtained, which includes multiple first video frames and original video frames corresponding to the first video frames; wherein the pixel values ​​of the target region image in the first video frame are preset pixel values; the first video frame is reconstructed using an initial fuzzing model to obtain a reconstructed video frame, and the reconstructed video frame is processed to obtain a corresponding desensitized block; a corresponding predicted video frame is obtained based on the desensitized block and the reconstructed video frame; a reconstruction loss is obtained based on the original video frame and the predicted video frame; features are extracted from the reconstructed video frame and the original video frame using the initial fuzzing model to obtain a first feature map corresponding to the reconstructed video frame and a second feature map corresponding to the original video frame; a feature matching loss is obtained based on the first feature map and the second feature map; the reconstructed video frame is encoded using the initial fuzzing model to obtain the pixel distribution of the encoded data of the reconstructed video frame; a KL divergence loss is obtained based on the pixel distribution and the normal distribution; a first loss is obtained based on the reconstruction loss, the feature matching loss, and the KL divergence loss; the model parameters of the initial fuzzing model are adjusted based on the first loss to obtain a trained fuzzing model.

[0009] Based on the aforementioned technical means, an initial blurring model is trained using video frames with pixel values ​​of 0 in multiple target regions. A first loss is obtained by calculating reconstruction loss, feature matching loss, and KL divergence loss. The model parameters of the initial blurring model are adjusted using the first loss until the convergence condition is met, thus obtaining a trained blurring model. The key video frames are reconstructed based on the trained blurring model to obtain reconstructed video frames. The reconstructed video frames are then blurred to obtain desensitized video frames with similar global features and local details to the key video frames, thereby improving the accuracy of the desensitized video frames.

[0010] In some embodiments, each video frame in the original video stream is verified, and video frames that meet preset requirements are determined as target video frames; feature information of the target video frames in the original video stream is obtained; the feature information in the target video frames is analyzed, and target video frames corresponding to feature information that meet preset requirements are determined as key video frames.

[0011] Based on the above technical means, the video frames in the original video stream are verified, and the target key frames are obtained by filtering the video frames in the original video stream based on the verification results. The feature information in the target key frames is extracted based on principal component analysis and machine learning methods. At the same time, the target video frames are filtered by combining the temporal and spatial features of the feature information of the target video frames to obtain key video frames, thereby improving the efficiency of key video frame filtering.

[0012] In some embodiments, the detection result includes the target detection score of the key video frame; the perturbation detection of the key video frame and the filtering of the key video frame based on the detection result include: performing pixel perturbation detection on the key video frame to obtain a first detection score; performing forgery perturbation detection on the key video frame to obtain a second detection score; obtaining a target detection score based on the first detection score and the second detection score, and acquiring key video frames whose target detection score is less than a preset score.

[0013] Based on the above technical means, by detecting deepfake samples in key video frames, key video frames are screened based on the detection scores, thereby reducing the workload of subsequent blurring processing and improving the efficiency of subsequent blurring processing.

[0014] In some embodiments, the key video frame is latent space encoded and decoded to obtain a reconstructed video frame of the key video frame; a reconstruction error and probability divergence characterizing the category of the key video frame are determined based on the reconstructed video frame and the key video frame; and a first detection score is obtained based on the reconstruction error and probability divergence.

[0015] Based on the above technical means, the corresponding reconstructed video frame is obtained by latent space encoding and decoding of key video frames, so as to reduce the impact of noise of key video frames on subsequent detection. The first detection score of the key video frame is determined based on the reconstruction error and probability divergence between the reconstructed video frame and the key video frame, and it is determined whether the key video frame is an adversarial sample, thereby increasing the workload of subsequent blurring processing and improving the efficiency of blurring processing.

[0016] In some embodiments, a second sample dataset is obtained, which includes multiple second video frames and a category label for each second video frame; wherein the second video frame includes a target object; the second video frame is latent space encoded and decoded using an initial adversarial example detection model to obtain a reconstructed image corresponding to the second video frame; the predicted category of the second video frame is obtained based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; a second loss is obtained based on the predicted category and category label of the second video frame; the model parameters of the initial adversarial example detection model are adjusted based on the second loss to obtain a trained adversarial example detection model.

[0017] Based on the above technical means, the trained adversarial example detection model is used to detect key video frames to determine whether they are adversarial examples, thereby filtering key video frames to reduce subsequent blurred processing data and improve the accuracy of blurring processing.

[0018] In some embodiments, feature extraction is performed on the key video frame to obtain a first feature map of the key video frame; the first feature map is resized to obtain a second feature map; the second feature map includes the features in the first feature map; a predicted value for the category of the key video frame is obtained based on the second feature map; and a second detection score is determined based on the predicted value.

[0019] Based on the aforementioned technical means, the second feature map of the key video frame is analyzed to obtain a predicted value for the second feature map. If the predicted value is greater than a preset threshold, the key video frame is characterized as a region forgery sample. The predicted value and the preset threshold are values ​​between 0 and 1. A predicted value greater than the preset threshold indicates that the key video frame is a region forgery sample, while a predicted value less than the preset threshold indicates that the key video frame is not a region forgery sample.

[0020] In some embodiments, a third sample dataset is obtained, which includes multiple third video frames and a category label for each third video frame; wherein the third video frame includes a target object; the third video frame is processed based on an initial fake sample detection model to obtain a predicted category; a third loss is obtained based on the predicted category and category label; the model parameters of the initial fake sample detection model are adjusted based on the third loss to obtain a trained fake sample detection model.

[0021] Based on the above technical means, the trained forgery sample detection model is used to detect key video frames to determine whether the key video frames are regional forgery samples, thereby filtering the key video frames to reduce the amount of data to be blurred in subsequent processing and thus improve the accuracy of blurring processing.

[0022] Secondly, embodiments of this application provide an image processing apparatus, comprising: an acquisition module for acquiring at least one key video frame from an original video stream; the key video frame being a video frame containing a target object; an identification module for identifying the target region where the target object is located in the key video frame; and a processing module for blurring the target region to obtain a desensitized video frame; wherein the image of the blurred region in the desensitized video frame is generated based on the image of the surrounding region of the target region.

[0023] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program that can run on the processor, and the processor executes the program to implement some or all of the steps in the above-described method.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0025] The beneficial effects of this application are as follows: by performing frame extraction on the original video stream to obtain key video frames containing the target object, the workload of subsequent blurring processing is reduced, thereby reducing resource consumption; by generating desensitized blocks corresponding to the target region through the features of the surrounding images of the target region, the target region is blurred through the desensitized blocks to obtain desensitized video frames with similar global features, local features and details to the corresponding key video frames, thereby improving the robustness, effectiveness and accuracy of the desensitized video frames.

[0026] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0027] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. Similar reference numerals with different letter suffixes may indicate different examples of similar parts. The drawings illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0028] Figure 1 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0029] Figure 2 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0030] Figure 3 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0031] Figure 4 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0032] Figure 5 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0033] Figure 6 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0034] Figure 7 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0035] Figure 8 A schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application;

[0036] Figure 9 A schematic diagram illustrating the implementation process of a keyframe recognition method provided in an embodiment of this application;

[0037] Figure 10 A flowchart illustrating a deepfake sample detection method provided in this application embodiment;

[0038] Figure 11 This is a schematic diagram of the structure of an adversarial sample detection branch provided in an embodiment of this application;

[0039] Figure 12 This is a schematic diagram of the structure of a regional forged sample detection branch provided in an embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the structure of a deep neural network provided in an embodiment of this application;

[0041] Figure 14 This is a schematic diagram of the composition structure of an image processing device provided in an embodiment of this application;

[0042] Figure 15 This is a schematic diagram of the hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application will be provided in conjunction with the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this application.

[0044] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the specific embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the disclosure of the present application to those skilled in the art.

[0045] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be practiced without one or more of these details. In other instances, to avoid confusion with this application, some technical features well-known in the art have not been described; that is, not all features of actual embodiments are described herein, nor are well-known functions and structures described in detail.

[0046] In the accompanying drawings, for clarity, the dimensions of layers, areas, and elements, as well as their relative dimensions, may be exaggerated. The same reference numerals denote the same elements throughout.

[0047] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising” and / or “including,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The technical and scientific terms used herein are for the purpose of describing embodiments of this application only and do not limit the scope of this application.

[0049] In related technologies, desensitization algorithms typically employ methods such as mosaic processing, occlusion processing, and obfuscation processing. Mosaic processing is a simple and intuitive method that achieves a blurring effect by replacing pixels within a region with surrounding pixel blocks of the same value. Occlusion processing usually involves covering a specified area with a pattern of a specific color or texture to protect privacy. Obfuscation processing methods typically involve transforming the color, brightness, or shape of a specified area to preserve its features. All of these methods directly convert surrounding pixels into desensitized pixels or paste image blocks of the same color or texture, inevitably resulting in the loss of important image features and detail in the feature map.

[0050] This application provides an image processing method that extracts frames from the original video stream to obtain key video frames containing the target object, thereby reducing the workload of subsequent blurring processing and reducing resource consumption. By generating desensitized blocks corresponding to the target region based on the features of the surrounding images, the target region is blurred using the desensitized blocks to obtain desensitized video frames with similar global features and local details to the corresponding key video frames, thereby improving the accuracy of the desensitized video frames.

[0051] Figure 1 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps S101 to S103, combining... Figure 1 The steps shown are explained.

[0052] Step S101: Obtain at least one key video frame from the original video stream.

[0053] The key video frame is a video frame containing the target object.

[0054] In some embodiments, the raw video stream may be video stream data of a real scene captured by a hardware camera.

[0055] In some embodiments, the target object may be a sensitive object such as a pedestrian's face or a license plate.

[0056] In some embodiments, after acquiring raw video stream data based on a hardware camera, the raw video stream first needs to be processed by frame extraction to obtain key video frames in the raw video stream that include the target object.

[0057] Step S102: Identify the target area where the target object is located in the key video frame.

[0058] In some embodiments, a trained object detection model can be used to identify the target region where the target object is located in the key video frame.

[0059] The training process of the trained object detection model includes: acquiring a sample dataset, which includes multiple video frames and the bounding boxes of the target objects in each video frame; inputting the multiple video frames into the object detection model to be trained to obtain the predicted detection boxes for the target objects; obtaining the loss value based on the bounding boxes of the target objects and the predicted bounding boxes; and adjusting the model parameters of the object detection model to be trained based on the loss value until the convergence condition is met, thereby obtaining the trained object detection model.

[0060] In some embodiments, the object detection model may include a convolutional neural network model (Visual GeometryGroup Network, VGG), a real-time object detection model (You Only Look Once, YOLO), a single-stage object detection model (Single Shot MultiBox Detector, SSD), etc., and this application does not specifically limit it.

[0061] Step S103: Blur the target area to obtain a desensitized video frame.

[0062] The image of the blurred area in the desensitized video frame is generated based on the image of the area surrounding the target area.

[0063] In some embodiments, the image of the surrounding area refers to the image of all areas outside the target area in the key video frame.

[0064] In some embodiments, an image of the area surrounding the target region in a key video frame is acquired, features of the surrounding area image are extracted, pixel blocks of the target region are generated based on the features of the surrounding area image, and the generated pixel blocks are used to blur the target region to obtain a desensitized video frame. It can be understood that the generated pixel blocks are pasted onto the target region to obtain a desensitized video frame.

[0065] The features of the surrounding area of ​​the image can be pixel information of the area, including features such as color, texture, edges, and shape.

[0066] In some embodiments, after obtaining the desensitized video frame, the method further includes obtaining a desensitized video stream based on the desensitized video frame and the original video stream.

[0067] In some embodiments, unblurred video frames from the original video stream are obtained, and based on the temporal sequence of the original video stream, the desensitized video frames and the unblurred video frames are synthesized to obtain a desensitized video stream.

[0068] In this embodiment, frame extraction is performed on the original video stream to obtain key video frames containing the target object, thereby reducing the workload of subsequent blurring processing and reducing resource consumption. Desensitized blocks corresponding to the target area are generated by using features of the surrounding images of the target area, and the target area is blurred by the desensitized blocks to obtain desensitized video frames with similar global features and local details to the corresponding key video frames, thereby improving the accuracy of the desensitized video frames.

[0069] Figure 2 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application. This method can be executed by a processor of an electronic device. Based on... Figure 1 , Figure 1 Step S103 can be updated to steps S201 to S203, combining Figure 2 The steps shown are explained.

[0070] Step S201: Reconstruct the key video frames to obtain reconstructed video frames.

[0071] In some embodiments, the target region pixels in the key video frame are set to 0 based on image processing software, the key video frame with 0 pixels in the target region image is latently encoded to obtain the latent space representation of the key video frame, and the latent space representation is decoded to obtain the reconstructed video frame.

[0072] Step S202: Obtain a desensitized block based on the surrounding image of the target region in the reconstructed video frame.

[0073] In some embodiments, features of the surrounding images of the target region in the reconstructed video frame are obtained, and a desensitization block for the target region is generated based on the features of the surrounding images of the target region, wherein the desensitization block has features similar to the surrounding images.

[0074] For example, a desensitized block of the same size as the target region is generated based on features such as color, texture, shape, and edges of the surrounding image of the target region.

[0075] Step S203: Obtain the desensitized video frame based on the reconstructed video frame and the desensitized block.

[0076] In some embodiments, the desensitized block obtained above is pasted into the target area of ​​the reconstructed video frame to obtain the desensitized video frame corresponding to the key video frame.

[0077] In this embodiment, latent space encoding and decoding are performed on key video frames to obtain reconstructed video frames with low noise. Based on the features of the surrounding images of the target region in the reconstructed video frame, a desensitized block for the target region is generated so that the generated desensitized video frame and its corresponding key video frame have similar global features and local details, preserving the information entropy of the entire key video frame, thereby improving the accuracy of the desensitized video frame.

[0078] In some embodiments, the process of obtaining the desensitized video frames described above is implemented through a trained blurring model, the training process of which includes:

[0079] Obtain a first sample dataset, which includes multiple first video frames and the original video frame corresponding to each first video frame; wherein the pixel value of the target region image in the first video frame is a preset pixel value.

[0080] In some embodiments, the target area of ​​the original video frame corresponding to the first video frame contains a target object; wherein, the target object in the original video frame may be a sensitive object, such as a face, license plate, location, etc., or it may be natural scenery, such as mountains, rivers, buildings, etc.

[0081] The first video frame is reconstructed using an initial fuzzing model to obtain a reconstructed video frame. The reconstructed video frame is then processed to obtain a corresponding desensitized block. Based on the desensitized block and the reconstructed video frame, a corresponding predicted video frame is obtained. The reconstruction loss is then obtained based on the original video frame and the predicted video frame.

[0082] In some embodiments, the reconstruction loss is obtained based on the mean square error of the pixels of the reconstructed video frame and the mean square error of the pixels of the predicted video frame, as shown in formula (1).

[0083]

[0084] Among them, L recon For reconstruction loss, x i To reconstruct the value of the i-th pixel in the video frame, To predict the value of the i-th pixel in a video frame, N is the total number of pixels in the reconstructed video frame.

[0085] The initial fuzzing model is used to extract features from the reconstructed video frame and the original video frame, resulting in a first feature map corresponding to the reconstructed video frame and a second feature map corresponding to the original video frame. The feature matching loss is then obtained based on the first and second feature maps.

[0086] In some embodiments, the first feature root map of the reconstructed video frame and the second feature map of the predicted video frame are extracted based on the feature extraction network layer in the initial fuzzing model. The feature matching loss is obtained based on the mean square error of the first feature map and the second feature map, as shown in formula (2).

[0087]

[0088] Among them, L feture For feature matching loss, This represents the feature map extracted by the j-th feature extraction network layer from the reconstructed video frame. Let M represent the feature map extracted by the j-th feature extraction network layer in the predicted video frame, and M represent the total number of feature maps.

[0089] The reconstructed video frames are encoded using an initial fuzzing model to obtain the pixel distribution of the encoded data of the reconstructed video frames; the KL divergence loss is obtained based on the pixel distribution and the normal distribution.

[0090] In some embodiments, the KL divergence loss is obtained based on the KL divergence and normal distribution of the coded data of the reconstructed video frame, as shown in formula (3).

[0091]

[0092] Among them, L KL For KL divergence loss, This represents the mean of the latent space representation of the reconstructed video frame. N represents the variance of the latent space representation of the reconstructed video frame. z This represents the dimension of the latent space representation of the reconstructed video frame.

[0093] The first loss is obtained based on the reconstruction loss, feature matching loss, and KL divergence loss; the model parameters of the initial fuzzy processing model are adjusted based on the first loss to obtain the trained fuzzy processing model.

[0094] In some embodiments, a first loss is obtained based on the reconstruction loss, feature matching loss, and KL divergence loss, as well as their respective preset weights, as shown in formula (4).

[0095] L mask =λ recon *L recon +λ feature *L feature +λ KL *L KL Formula (4)

[0096] Among them, L mask For the first loss, λ recon λ is the preset weight for the reconstruction loss. feature λ is the preset weight for the feature matching loss. KL Preset weights for KL divergence loss.

[0097] The first loss is obtained based on the reconstruction loss, feature matching loss, and KL divergence loss; the model parameters of the initial fuzzy processing model are adjusted based on the first loss to obtain the trained fuzzy processing model.

[0098] In some embodiments, the model parameters of the initial fuzzy processing model are adjusted based on the first loss until the model parameters meet the convergence condition, and the trained fuzzy processing model is output.

[0099] In this embodiment, an initial blurring model is trained using video frames with pixel values ​​of 0 in multiple target regions. A first loss is obtained by calculating reconstruction loss, feature matching loss, and KL divergence loss. The model parameters of the initial blurring model are adjusted using the first loss until the convergence condition is met, thereby obtaining a trained blurring model. The key video frames are reconstructed based on the trained blurring model to obtain reconstructed video frames. The reconstructed video frames are then blurred to obtain desensitized video frames with similar global features and local details to the key video frames, thereby improving the accuracy of the desensitized video frames.

[0100] Figure 3 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application. The method can be executed by a processor of an electronic device. The flow may include steps S301 to S303, combining... Figure 3 The steps shown are explained.

[0101] Step S301: Verify each video frame in the original video stream, and determine the video frames that meet the preset requirements as target video frames.

[0102] In some embodiments, each video frame in the original video stream is subjected to Kaiser-Meyer-Olkin (KMO) and Bartlett's Test of Sphericity (BartLett) validation, including: setting the KMO metric value as x1, the approximate chi-square value of the BartLett test of sphericity as x2, and the significance level as x3; performing KMO and BartLett validation on each video frame to obtain the corresponding validation value for each video frame; if the validation value of a video frame satisfies x1>0.5, x2>3000, and x3<0.05, the video frame is considered to meet the preset requirements, thereby identifying the video frame as the target video frame; wherein, the target video frame represents a video frame from which features can be extracted.

[0103] Step S302: Obtain the feature information of the target video frame in the original video stream.

[0104] In some embodiments, the feature information of the target video frame includes the feature vector and feature value of the image in the target video frame.

[0105] In some embodiments, the eigenvalues ​​and eigenvectors of the principal component factors in the target video frame are calculated based on multivariate data analysis.

[0106] For example, multivariate data analysis methods include principal component analysis (PCA). PCA is used to calculate the eigenvalues ​​and eigenvectors of the principal component factors in the target video frame. This includes: representing the target video frame as a matrix; preprocessing the target video frame, such as grayscale conversion, binarization, and dilation, to eliminate noise and interference; standardizing the matrix to make its mean 0 and variance 1; calculating the covariance matrix of the matrix to represent the correlation between pixels in the target video frame; and performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors. Here, the eigenvalues ​​represent the variance of the principal component factors, i.e., the degree to which the principal component factors explain the variation in the dataset; and the eigenvectors represent the direction of the principal component factors.

[0107] Step S303: Analyze the feature information in the target video frame, and determine the target video frame corresponding to the feature information that meets the preset requirements as the key video frame.

[0108] In some embodiments, the feature values ​​and feature vectors of the target video frames are analyzed, and the target video frames are filtered based on the magnitude of the target feature values ​​and the direction of the feature vectors to obtain candidate video frames. If the number of feature values ​​and feature vectors exceeds a threshold, the feature values ​​and feature vectors are predicted based on the trained first image recognition model to obtain a prediction contribution rate. The target video frames corresponding to feature values ​​and feature vectors with contribution rates reaching a preset threshold are determined as candidate video frames.

[0109] In some embodiments, the training process of the first image recognition model after training includes: acquiring a sample dataset, the sample data including feature vectors and feature values ​​of multiple video frames and a label indicating whether each video frame is a key video frame; inputting the feature vectors and feature values ​​of the multiple video frames into an initial first image recognition model to obtain the predicted contribution rate of the feature vectors and feature values ​​characterizing whether a video frame is a key video frame; and adjusting the model parameters of the initial first image recognition model based on the loss value of the predicted contribution rate and the label to obtain the trained first image recognition model. The first image recognition model includes a fully connected layer and a Softmax layer.

[0110] In some embodiments, since the video stream is temporally continuous and adjacent frames are close and continuous, to better capture temporal information, model based on long-term dependencies, and utilize contextual information, temporal analysis is performed on target video frames including principal component factors. This includes predicting the principal component factors of candidate video frames based on a trained second image recognition model, thereby identifying candidate video frames corresponding to the principal component factors that satisfy the temporal characteristics of the original video stream as key video frames. The performance of the key video frames obtained from the candidate video frames is then evaluated based on the identified key video frames to determine the accuracy of the trained second image recognition model. The second image recognition model includes fully connected layers and a Softmax layer.

[0111] In this embodiment, the video frames in the original video stream are verified, and the target key frames are obtained by filtering the video frames in the original video stream based on the verification results. The feature information in the target key frames is extracted based on the principal component analysis and machine learning methods. At the same time, the target video frames are filtered by combining the temporal and spatial features of the feature information of the target video frames to obtain the key video frames, thereby improving the efficiency of key video frame filtering.

[0112] Figure 4 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application. This method can be executed by a processor of an electronic device. Based on... Figure 1 , Figure 1 Step S101 may be followed by step S401, which combines... Figure 4 The steps shown are explained.

[0113] Step S401: Perform disturbance detection on the key video frames, and filter the key video frames based on the detection results.

[0114] In some embodiments, deepfake sample detection is performed on key video frames to filter out key video frames containing deepfake samples, while retaining the original clean key video frames.

[0115] Among them, deepfake samples include adversarial samples and region-limited fake samples. Adversarial samples represent pixel-level perturbations that are indivisible to the human eye and are superimposed on clean samples. Region-limited fake samples usually refer to samples that exist in the real physical world in the form of patches. Adversarial samples and region-limited fake samples can mislead neural network-based target detection models to make incorrect predictions.

[0116] Figure 5 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application. This method can be executed by a processor of an electronic device. Based on... Figure 4 The detection results include the target detection scores of the key video frames; Figure 4 Step S401 can be updated to steps S501 to S503, combining Figure 5 The steps shown are explained.

[0117] Step S501: Perform pixel perturbation detection on the key video frame to obtain a first detection score.

[0118] In some embodiments, key video frames are reconstructed to obtain reconstructed video frames. Based on the pixels of the reconstructed video frames and the pixels of the key video frames, the reconstruction loss of the reconstructed video frames and the key video frames is obtained. When the reconstruction loss reaches a preset threshold, the key video frames are characterized as adversarial examples.

[0119] In some embodiments, the probabilistic divergence between the reconstructed video frame and the key video frame is obtained based on the Jensen-Shannon (JS) divergence or Kullback-Leibler (KL) divergence of the key video frame and the JS divergence or KL divergence of the reconstructed video frame. When the probabilistic divergence reaches a preset threshold, the key video frame is characterized as an adversarial example.

[0120] In some embodiments, a first detection score is obtained based on reconstruction loss and probabilistic divergence.

[0121] Step S502: Perform forgery perturbation detection on the key video frames to obtain a second detection score.

[0122] In some embodiments, feature extraction is performed on key video frames to obtain feature maps of key video frames, and the feature maps of key video frames are analyzed to obtain predicted values ​​of key video frames. If the predicted values ​​reach a preset threshold, the key video frames are determined to be region-defined forgery samples, and a second detection score is determined based on the predicted values.

[0123] Step S503: Obtain the target detection score based on the first detection score and the second detection score, and acquire the key video frames whose target detection score is less than a preset score.

[0124] In some embodiments, the target detection score is obtained based on the first detection score, the second detection score, and the weights of the first detection score and the second detection score, as shown in formula (5).

[0125] F(x)=τ*F1(x)+(1-τ)*F2(x) Formula (5)

[0126] Where F(x) is the target detection score, F1(x) is the first detection score, F2(x) is the second detection score, τ is the weight of the first detection score, and (1-τ) is the weight of the second detection score.

[0127] For example, if the first detection score is F1(x) = [α1,β1] and the second detection score is F2(x) = [α2,β2], then the target detection score is F(x) = [τ·α1 + (1-τ)·α2τβ1 + (1-τ)·β2].

[0128] In this embodiment of the application, by detecting deepfake samples on key video frames, key video frames are filtered based on target detection scores, thereby reducing the workload of subsequent blurring processing and improving the efficiency of subsequent blurring processing.

[0129] Figure 6 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application. This method can be executed by a processor of an electronic device. Based on... Figure 5 , Figure 5 Step S501 can be updated to steps S601 to S603, combining Figure 6 The steps shown are explained.

[0130] Step S601: Perform latent space encoding and decoding on the key video frame to obtain the reconstructed video frame of the key video frame.

[0131] In some embodiments, the key video frame is latent space encoded to obtain a latent space representation of the key video frame, and the latent space representation is decoded to obtain a reconstructed video frame of the key video frame.

[0132] Step S602: Determine the reconstruction error and probability divergence that characterize the category of the key video frame based on the reconstructed video frame and the key video frame.

[0133] In some embodiments, the reconstruction loss of the reconstructed video frame and the key video frame is obtained based on the pixels of the reconstructed video frame and the pixels of the key video frame. When the reconstruction loss reaches a preset threshold, the key video frame is characterized as an adversarial example.

[0134] In some embodiments, the probability divergence between the reconstructed video frame and the key video frame is obtained based on the JS divergence or KL divergence of the key video frame and the JS divergence or KL divergence of the reconstructed video frame. When the probability divergence reaches a preset threshold, the key video frame is characterized as an adversarial example.

[0135] Step S603: Obtain the first detection score based on the reconstruction error and probability divergence.

[0136] In some embodiments, the weighted sum of the reconstruction error and the probability divergence is used as the first detection score based on the weights corresponding to the reconstruction error and the probability divergence.

[0137] In this embodiment, key video frames are reconstructed by performing latent space encoding and decoding on them to reduce the impact of noise on subsequent detection. Based on the reconstruction error and probability divergence between the reconstructed video frames and the key video frames, the first detection score of the key video frame is determined and it is determined whether the key video frame is an adversarial sample, thereby reducing the workload of subsequent blurring and improving the efficiency of blurring.

[0138] In some embodiments, the reconstruction error and probability divergence used to determine the category of the key video frame are implemented through an adversarial example detection model, the training process of which includes:

[0139] Obtain a second sample dataset, which includes multiple second video frames and a category label for each second video frame; wherein, the second video frames include the target object.

[0140] In some embodiments, the second video frame may be a historical video frame that includes sensitive objects, such as a video frame that includes faces, license plates, etc., or a video frame that includes natural scenery or buildings.

[0141] In some embodiments, the category label of the second video frame is a label that characterizes whether the second video frame is an adversarial example.

[0142] The second video frame is latent space encoded and decoded using the initial adversarial example detection model to obtain the reconstructed image corresponding to the second video frame; the predicted category of the second video frame is obtained based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; and the second loss is obtained based on the predicted category and category label of the second video frame.

[0143] In some embodiments, if the reconstruction loss is greater than a preset threshold, the second video frame is characterized as an adversarial example; if the probability divergence is greater than a preset threshold, the second video frame is characterized as an adversarial example.

[0144] The model parameters of the initial adversarial example detection model are adjusted based on the second loss to obtain the trained adversarial example detection model.

[0145] In some embodiments, the thresholds for reconstruction loss and probability divergence in the initial adversarial example detection model are adjusted based on the second loss until the convergence condition is met, and the trained adversarial example detection model is output.

[0146] In this embodiment of the application, a trained adversarial example detection model is used to detect key video frames to determine whether they are adversarial examples, thereby filtering key video frames to reduce subsequent blurred processing data and improve the accuracy of blurring processing.

[0147] Figure 7 This is a schematic diagram illustrating the implementation flow of an image processing method provided in an embodiment of this application. This method can be executed by a processor of an electronic device. Based on... Figure 5 , Figure 5 Step S502 can be updated to steps S701 to S704, combining Figure 7 The steps shown are explained.

[0148] Step S701: Extract features from the key video frame to obtain the first feature map of the key video frame.

[0149] In some embodiments, key video frames of the first size are preprocessed, including denoising, frame rate adjustment, color correction, etc., and feature extraction is performed on the preprocessed key video frames to obtain a first feature map of the first size.

[0150] Step S702: Perform a size transformation on the first feature map to obtain a second feature map; the second feature map includes the features in the first feature map.

[0151] In some embodiments, the first feature map is resized to obtain a second feature map of a preset size.

[0152] Step S703: Obtain the predicted value of the category for the key video frame based on the second feature map.

[0153] In some embodiments, a second feature map of a key video frame is analyzed to obtain a predicted value for the second feature map. If the predicted value is greater than a preset threshold, the key video frame is characterized as a region forgery sample. The predicted value and the preset threshold are values ​​between 0 and 1. A predicted value greater than the preset threshold indicates that the key video frame is a region forgery sample, and a predicted value less than the preset threshold indicates that the key video frame is not a region forgery sample.

[0154] Step S704: Determine the second detection score based on the predicted value.

[0155] In some embodiments, the predicted value is used as the second detection score.

[0156] In this embodiment, a first feature map of corresponding size is obtained by extracting features from key video frames. The first feature map is then transformed to obtain a second feature map that meets the analysis requirements. The second feature map is then analyzed to obtain a predicted value for the category of the key video frame. Based on the predicted value, it is determined whether the key video frame is a region forgery sample, thereby filtering the key video frames to reduce subsequent blurred processing data and improve the accuracy of blurring processing.

[0157] In some embodiments, the process of obtaining the predicted value of the category for the key video frame is implemented by a trained forgery detection model, the training process of which includes:

[0158] Obtain a third sample dataset, which includes multiple third video frames and a category label for each third video frame; wherein, the third video frame includes the target object.

[0159] In some embodiments, the third video frame may be a historical video frame that includes sensitive objects, such as a video frame that includes faces, license plates, etc., or a video frame that includes natural scenery or buildings.

[0160] In some embodiments, the category label of the third video frame is a label that characterizes whether the third video frame is a region forgery sample.

[0161] The third video frame is processed based on the initial fake sample detection model to obtain the predicted category; the third loss is obtained based on the predicted category and category label.

[0162] In some embodiments, features are extracted from the third video frame based on the initial forgery detection model to obtain a first feature map corresponding to the size of the third video frame. The first feature map is then resized to obtain a second feature map of a size that the initial forgery detection model can analyze. The second feature map is then analyzed to obtain a predicted value characterizing whether the third video frame is a region forgery sample.

[0163] The model parameters of the initial fake sample detection model are adjusted based on the third loss to obtain the trained fake sample detection model.

[0164] In some embodiments, the model parameters of the initial fake sample detection model are adjusted based on the third loss adjustment until the convergence condition is met, and the trained fake sample detection model is output.

[0165] In this embodiment of the application, a trained forgery sample detection model is used to detect key video frames to determine whether the key video frames are forged samples of a region, thereby filtering the key video frames to reduce the amount of data to be blurred in subsequent processing and thus improve the accuracy of blurring processing.

[0166] The following describes an exemplary application of the image processing method provided in this application embodiment in a real-world scenario.

[0167] With increasing user emphasis on privacy protection, relevant policies and regulations require the anonymization of potentially privacy-leaking real data (such as facial and palm print biometrics) collected by sensor hardware or camera devices. Adherence to anonymization principles during data processing is encouraged, with sensitive data being anonymized and de-identified. In the field of intelligent vehicles, map-based autonomous driving technology typically requires perception modeling of real roads. This involves storing and processing real data collected by data collection vehicles via cameras on the road, as well as optimizing parameters for deep learning model training. Due to limited vehicle data storage space and computing power, online training of instantly collected road data is not supported. The current common solution is to upload the data to the cloud for training and parameter optimization on large models on servers. Relevant policies and regulations require companies to legally handle facial and license plate data from driving videos to protect public safety. Currently, to meet user privacy protection goals, intelligent vehicles typically anonymize facial and license plate information before offline uploading to the cloud for large-scale model training and parameter optimization. The anonymization algorithm includes two steps: locating the area to be anonymized and blurring. De-identification region localization typically relies on detection models based on text recognition or image recognition to locate the object to be de-identified in the video stream, and then determines the corresponding coordinate bounding box based on the located object. Blurring processing usually involves mosaic processing or image replacement based on transfer learning methods.

[0168] In related technologies, video stream anonymization methods directly perform region detection and blurring on the data collected by camera hardware. Region detection directly locates regions in the collected video stream data, while blurring is based on the coordinates of the rectangle to be anonymized. If the rectangle's location is abnormal, it can lead to user privacy leaks. Furthermore, incorrect anonymization of other regions can mislead the neural network into abnormal classification, resulting in the loss of data value and resources. In real-world road conditions, deepfake samples have the ability to cause target detection models to lose sight of their targets. With the continuous improvement and evolution of existing deep learning technologies, adversarial examples and region-defined forgeries in real-world scenarios can lead to incorrect predictions of the target region to be anonymized. Adversarial examples refer to the superposition of adversarial perturbations that are indistinguishable to the human eye onto clean samples, thereby inducing the neural network to misclassify. In the field of intelligent vehicles, region-defined forgeries superimpose perturbation blocks that are visible to the human eye onto license plates, traffic signs, or traffic lights. Although visible to the human eye, these perturbation blocks produce abnormal processing results for autonomous vehicles. Due to the existence of adversarial examples and region-defined forgeries, the regions to be anonymized become abnormal, causing the anonymization algorithm to err on its part by either not anonymizing the original video stream or performing abnormal anonymization, thus leading to privacy leaks.

[0169] In this embodiment, before desensitizing the video stream, deepfake sample detection is performed on the input video frames, including the identification of adversarial samples and region-specific fake samples. The video stream desensitization method based on deep learning security can filter out abnormal samples, improve the robustness of region localization, and thus improve the quality of the desensitized video. At the same time, in order to reduce resource consumption, only key frames of the video stream are processed, rather than all video frames.

[0170] In related technologies, desensitization algorithms typically include blurring, mosaicking, occlusion, and obfuscation. Blurring uses Gaussian discrete approximation, averaging, and median values ​​to smooth the image while preserving details, simplifying noise reduction, and eliminating abrupt noise. Mosaicking is a simple and intuitive method that replaces pixels within a region with surrounding pixels of the same value to achieve a blurring effect. Occlusion typically involves covering a specified region with a pattern of a specific color or texture to protect privacy. Obfuscation methods usually involve transforming the color, brightness, or shape of a specified region to preserve features. However, these methods almost entirely ignore the impact of the desensitized image on subsequent cloud-based large-scale model inference. Directly converting surrounding pixels into desensitized pixels or pasting image blocks of the same color or texture into the original image inevitably leads to the loss of important image features and detail in the feature map, thus affecting the detection and classification accuracy of subsequent models.

[0171] In this embodiment of the application, in order to ensure that the image with missing parts and the real image have similar feature maps and details, a pre-trained VGG network is used to extract the features of the image, which includes multiple convolutional layers and pooling layers; the similarity between the repaired image and the original image is maintained by minimizing the feature space distance between them.

[0172] In the embodiments of this application, the deep learning security protection for key video frames can achieve robust performance in locating the region to be desensitized during video stream desensitization, while maintaining performance stability and reducing resource consumption.

[0173] Figure 8 This is a schematic flowchart illustrating the implementation of an image processing method according to an embodiment of this application. The method can be executed by a processor of an electronic device. The flowchart includes steps S801 to S804, which combine... Figure 8 The steps shown are explained.

[0174] Step S801: For the raw video stream data acquired by the camera hardware, key video frames are obtained by performing frame extraction processing on the raw video stream.

[0175] In some embodiments, if the input raw video stream data is encoded, then the raw video stream data needs to be decoded first.

[0176] In some embodiments, the raw input video stream is first subjected to frame extraction to reduce resource consumption. In the field of intelligent vehicles, front-view and rear-view cameras typically acquire more and more important data, thus requiring less frame extraction and resulting in a higher frame rate for the video stream compared to other non-front-view and rear-view cameras. Simultaneously, the priority of surround-view and panoramic cameras is adaptively adjusted based on the application scenario, with higher-priority cameras capturing video streams at higher frame rates. Typically, the video stream frame rate is maintained within the range of 40 frames per second.

[0177] Figure 9 This is a schematic diagram illustrating the implementation flow of a keyframe recognition method provided in an embodiment of this application. This method can be executed by a processor of an electronic device. The method includes steps S901 to S904, combining... Figure 9 The steps shown are explained.

[0178] Step S901: Verify the original video stream data.

[0179] In some embodiments, the raw video stream data may be historical video frames or video frames from the training set.

[0180] In some embodiments, it is necessary to consider performing KMO and Bart Lett verification on historical video frames or video frames in the training set. Specifically, this includes setting the KMO metric value to x1, the approximate chi-square value of the Bart Lett spherical verification value to x1, and the significance level to x3. In actual detection, in the embodiments of this application, when x1>0.5, x2>3000, and x3<0.05, factor analysis can be performed on the video frame data to identify key frames of the original video stream data.

[0181] Step S902: Extract feature information from the original video stream based on principal component analysis.

[0182] In some embodiments, the feature information includes feature vectors and feature values.

[0183] In some embodiments, principal component analysis (PCA) is used to calculate the eigenvectors and eigenvalues ​​of factors from historical video frames or video stream data in the training set. Specifically, PCA-based dimensionality reduction data samples are selected based on the variance contribution rate of the factors to extract the main influencing factors and improve the accuracy of their weights.

[0184] Step S903: Filter the feature information based on the fully connected layer and Softmax layer of the neural network to obtain the principal component factors.

[0185] In some embodiments, if the number of calculated feature vectors and feature values ​​is large, in order to improve the keyframe recognition speed, the feature vectors and feature values ​​obtained from principal component analysis are input into a cascaded structure of fully connected layers and softmax layers for filtering. This includes training the cascaded structure based on predetermined labels after converting them to one-hot encoding and then using the BP backpropagation algorithm. For a well-trained model, effective and reasonable feature vectors and feature values ​​are selected by comparing the predicted contribution rates to obtain effective and reasonable principal component factors.

[0186] Step S904: Perform RNN temporal analysis on the video frames including the main component factors to obtain the key video frames.

[0187] In some embodiments, since the video stream is temporally continuous and the content of adjacent frames is close and continuous, in order to better meet the requirements of temporal information capture, long-term dependency modeling, and contextual information utilization, temporal analysis is performed on video frames including major component factors. This includes: dividing historical video frames into training set, test set, and validation set in a ratio of 7:2:1; extracting temporal features including major component factors from key video frames in the training set; performing inference in the test set to obtain key video frames; and finally evaluating the performance of the obtained key video frames based on the validation set.

[0188] Step S802: Detect key video frames to obtain target key video frames.

[0189] In some embodiments, deepfake sample detection is performed on key video frames to filter out key video frames containing deepfake samples, while retaining the original clean key video frames as target key video frames.

[0190] In some embodiments, deepfake samples include adversarial examples and region-defined forgery samples. Adversarial examples refer to pixel-level perturbations that are indistinguishable to the human eye superimposed on clean samples. Region-defined forgery samples typically exist in the real physical world as patches. Adversarial examples and region-defined forgery samples can mislead neural network-based target detection models into making incorrect predictions, causing the bounding boxes of the object to be desensitized to be abnormal, thus negating the purpose of desensitization. To ensure the effective localization of the region to be desensitized, this application employs a dual-branch fusion deepfake sample detection method that combines adversarial example detection and region-defined forgery sample detection branches to remove key video frames containing adversarial examples and region-defined forgery samples.

[0191] Figure 10 This is a flowchart illustrating a deepfake sample detection method provided in an embodiment of this application. The method can be executed by a processor of an electronic device. The process includes steps S1001 to S1003, which will be combined with... Figure 10 The steps shown are explained.

[0192] Step S1001: Detect key video frames based on the adversarial example detection branch to obtain the first discrimination score.

[0193] In some embodiments, please refer to Figure 11 , Figure 11This application provides a schematic diagram of the structure of an adversarial example detection branch, including an encoder 1101, a decoder 1102, and a deep neural network 1103. The adversarial example detection branch is trained using historical video frames to obtain its detection threshold. The training process includes: inputting historical video frames as sample data into the adversarial example detection branch, and providing annotation information on whether the historical video frames are adversarial examples; mapping the historical video frames to a latent space using the encoder 1101 to obtain the latent space representation of the historical video frames; and then... Decoder 1102 maps the latent space representation of historical video frames to the original space to obtain reconstructed video frames. Based on deep neural network 1103, it obtains the reconstruction error between the historical video frames and the reconstructed video frames. Based on deep neural network 1103, it calculates the JS divergence or KL divergence to obtain the probabilistic divergence between the historical video frames and the reconstructed video frames. Based on the reconstruction error and probabilistic divergence, it predicts the historical video frames to obtain prediction results. Based on the prediction results and the loss value of the labeled information, it adjusts the parameters of deep neural network 1103 until the convergence condition is met, and outputs the trained adversarial example detection branch. The parameters include the weights for determining the first discrimination score based on the reconstruction error and probabilistic divergence; and the thresholds for determining whether a historical video frame is an adversarial example based on the reconstruction error and the probabilistic divergence.

[0194] In some embodiments, key video frames are input to a trained adversarial example detection branch to obtain the reconstruction error and probability divergence of the key video frames, as well as whether the key video frames are adversarial examples, and a first discrimination score is obtained based on the weights of the reconstruction error and probability divergence of the key video frames.

[0195] Step S1002: Detect key video frames based on the regional forgery sample detection branch to obtain the second discrimination score.

[0196] In some embodiments, the method for detecting region-limited forgery samples focuses on checking whether there are perturbation blocks in multiple local regions of an image, and therefore can be implemented using Convolutional Neural Networks (CNNs). CNNs focus on local image regions through local receptive fields, and then integrate different local receptive fields in deeper network layers to obtain overall image information. This method integrates feature extraction, selection, and classification into the entire model, improving feature representation ability and enhancing feature discriminability through end-to-end training. In this application embodiment, a binary classifier based on transfer learning is used to identify forgery samples. At the same time, in order to achieve high-precision adversarial discrimination of samples with various resolutions, an adaptive input resolution sample processing module is integrated into the binary classifier based on transfer learning. This module adds adaptive pooling layers and convolutional layers based on small-sized convolutional kernels while keeping the original network's feature extractor and feature classifier unchanged, thereby achieving the legitimacy discrimination of test samples with multiple input resolutions.

[0197] In some embodiments, the region forgery detection branch is trained based on historical video frames and annotation information indicating whether the historical video frames are region-defined forgery samples. The historical video frames are input into the region forgery detection branch to obtain predicted values ​​characterizing whether the historical video frames are region-defined forgery samples. The parameters of the region forgery detection branch are adjusted based on the loss of the predicted values ​​and annotation information until the convergence function is satisfied, and the trained region forgery detection branch is output.

[0198] In some embodiments, the annotation information can be label information of 0 and 1, where 0 indicates that the historical video frame is a region forged sample and 1 indicates that the historical video frame is a clean sample, and the predicted value can be a value in the range of 0 to 1.

[0199] In some embodiments, please refer to Figure 12 , Figure 12 The schematic diagram of a region forgery sample detection branch provided in this application embodiment includes a preprocessing module 1201 and a deep neural network module 1202. The preprocessing module 1201 is used to preprocess the key video frames obtained above so that the processed key video frames meet the processing requirements of the deep neural network model 1202. Based on the deep neural network model 1202, the processed key video frames are processed to obtain the weight parameters of the key video frames. Based on the weight parameters, it is determined whether the key video frame is a region forgery sample and a second discrimination score.

[0200] In some embodiments, please refer to Figure 13 , Figure 13This is a schematic diagram of a deep neural network structure provided in an embodiment of this application, including an input video frame 1301, a first feature map 1302, an adaptive pooling layer 1303, multiple small-sized convolutional kernels 1304, a second feature map 1305, and a fully connected layer 1306. First, features are extracted from the input video frames 1301 of different sizes to obtain M first feature maps 1302 of different sizes. The M first feature maps 1302 of different sizes are then input into the adaptive pooling layer 1303 and the multiple small-sized convolutional kernels 1304 for processing to obtain N second feature maps 1305 of the same size. The N second feature maps 1305 of the same size are then input into the fully connected layer 1306 to obtain the predicted value of the input video frame corresponding to each second feature map 1305.

[0201] Step S1003: Based on the first discrimination score and the second discrimination score, determine the target key video frame.

[0202] In some embodiments, a target discrimination score is obtained based on a first discrimination score and a second discrimination score. The key video frames with a target discrimination score greater than a preset score can be removed by referring to the above formula (1) to obtain the target key video frames.

[0203] Step S803: Identify the desensitized region of the target key video frame and blur the desensitized region to obtain the desensitized video frame.

[0204] In some embodiments, the desensitized region is a region that includes target features, such as facial information, license plate information, etc.

[0205] In some embodiments, directly converting surrounding pixels into desensitized pixels or pasting image patches of the same color or texture will inevitably result in the loss of important image features and detail in the feature maps, thus affecting the detection and classification accuracy of subsequent models. Therefore, to ensure that the image with missing pixels and the real image have similar feature maps and details, this embodiment of the application proposes to use a pre-trained deep neural network to extract features from the surrounding images of the area to be desensitized. The feature extraction network layer includes multiple convolutional layers and pooling layers. This embodiment aims to maintain the similarity between the repaired image and the original image by minimizing the feature space distance between them. Specifically, this embodiment uses a combination of reconstruction loss, feature matching loss, and KL divergence as the training loss function to generate desensitized patches, which are then pasted onto the area to be desensitized to maximize the preservation of the information entropy of the entire video frame for subsequent training and model inference.

[0206] In some embodiments, key video frames are input into a trained target detection model to identify desensitized regions of target features in the key video frames, and the desensitized regions are marked with detection boxes to obtain labeled video frames.

[0207] In some embodiments, preprocessed video frames are input into a trained blurring model to obtain desensitized video frames. The blurring model first sets the pixel values ​​of the desensitized regions to 0, and generates desensitized blocks in other regions based on the desensitized regions with pixel values ​​of 0 and adds them to the desensitized regions to obtain desensitized video frames. The desensitized video frames have similar global features and local details to the video frames in the original video stream.

[0208] In some embodiments, the fuzzing model consists of a variational autoencoder and a deep neural network. The training process of the trained fuzzing model includes: acquiring a sample dataset, which includes multiple historical video frames, with the desensitized region of the target area in each historical video frame labeled by a detection box; inputting the multiple historical video frames into the variational autoencoder to obtain multiple first video frames, whereby the variational autoencoder sets the pixel value of the desensitized region of each first video frame to 0; inputting the multiple first video frames into the fuzzing model to be trained to fill the desensitized region with pixels from the surrounding area to obtain a second video frame; obtaining the reconstruction loss of the first video frame and the second video frame; extracting the first feature map and the second feature map of the historical video frames to obtain the feature matching loss of the first feature map and the second feature map; encoding the historical video frames to obtain the latent space representation of the historical video frames; obtaining the pixel distribution of the latent space representation; and obtaining the KL divergence loss based on the pixel distribution and the normal distribution; obtaining the target loss based on the reconstruction loss, the feature matching loss, the KL divergence loss, and their respective weights; and adjusting the parameters of the fuzzing model to be trained based on the target loss to obtain the trained fuzzing model. The sample dataset can also use video frames that are unrelated to the target features, such as natural landscape images.

[0209] In some embodiments, the mean squared error is used as the reconstruction loss to measure the difference between the repaired image and the original image at the pixel level, referring to the above formula (1).

[0210] In some embodiments, for feature matching loss, features of the repaired image and the original image are extracted based on the VGG network as the feature extraction network, and the differences between them are compared. The feature map of a single layer can be selected to calculate the loss, or the feature maps of multiple layers can be weighted and combined. Mean squared error or other distance metrics can be used to measure the differences between features, as shown in formula (2) above.

[0211] For KL divergence loss, by constraining the pixel distribution expressed in the latent space of the input video frame to be close to a normal distribution, that is, a unit Gaussian distribution, the KL divergence loss is obtained, referring to the above formula (3).

[0212] In some embodiments, referring to the above formula (4), the total loss is obtained based on the reconstruction loss, feature matching loss, KL divergence loss and the corresponding preset weights, so as to train the fuzzy processing model.

[0213] Step S804: Obtain desensitized video stream data based on the desensitized target key video frames and the original video stream data.

[0214] In some embodiments, the desensitized video frames are combined with the original video frames and packaged into a complete desensitized video stream file, taking into account timing.

[0215] In this embodiment, frame extraction is performed on the original video stream to obtain key video frames containing the target object, thereby reducing the workload of subsequent blurring processing and reducing resource consumption. Desensitized blocks corresponding to the target area are generated by using features of the surrounding images of the target area, and the target area is blurred by the desensitized blocks to obtain desensitized video frames with similar global features and local details to the corresponding key video frames, thereby improving the accuracy of the desensitized video frames.

[0216] Figure 14 This is a schematic diagram of the composition structure of an image processing device provided in an embodiment of this application, as shown below. Figure 14 As shown, the image processing device 1400 includes: an acquisition module 1401, an identification module 1402, and a processing module 1403, wherein: the acquisition module 1401 is used to acquire at least one key video frame in the original video stream; the key video frame is a video frame with a target object; the identification module 1402 is used to identify the target area where the target object is located in the key video frame; the processing module 1403 is used to blur the target area to obtain a desensitized video frame; wherein the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

[0217] In some embodiments, the processing module 1403 is further configured to reconstruct the key video frame to obtain a reconstructed video frame; obtain a desensitized block based on the surrounding image of the target region in the reconstructed video frame; and obtain the desensitized video frame based on the reconstructed video frame and the desensitized block.

[0218] In some embodiments, the image processing device 1400 further includes a training module (not shown in the figure), configured to acquire a first sample dataset, the first sample dataset including multiple first video frames and original video frames corresponding to the first video frames; wherein the pixel values ​​of the target region image in the first video frame are preset pixel values; the first video frame is reconstructed using an initial blurring model to obtain a reconstructed video frame of the first video frame, and the reconstructed video frame is processed to obtain a corresponding desensitized block; a corresponding predicted video frame is obtained based on the desensitized block corresponding to the reconstructed video frame and the reconstructed video frame; and a reconstruction loss is obtained based on the original video frame and the predicted video frame. Feature extraction is performed on the reconstructed video frames and the original video frames using an initial fuzzing model to obtain a first feature map corresponding to the reconstructed video frames and a second feature map corresponding to the original video frames. A feature matching loss is obtained based on the first and second feature maps. The reconstructed video frames are then encoded using the initial fuzzing model to obtain the pixel distribution of the encoded data. A KL divergence loss is obtained based on the pixel distribution and the normal distribution. A first loss is obtained based on the reconstruction loss, feature matching loss, and KL divergence loss. The model parameters of the initial fuzzing model are adjusted based on the first loss to obtain the trained fuzzing model.

[0219] The acquisition module 1401 is further configured to verify each video frame in the original video stream, determine the video frame that meets the preset requirements as the target video frame; acquire the feature information of the target video frame in the original video stream; analyze the feature information in the target video frame, and determine the target video frame corresponding to the feature information that meets the preset requirements as the key video frame.

[0220] In some embodiments, the image processing apparatus 1400 further includes a detection module (not shown in the figure), used for the detection result including the target detection score of the key video frame; the step of performing perturbation detection on the key video frame and filtering the key video frame based on the detection result includes: performing pixel perturbation detection on the key video frame to obtain a first detection score; performing forgery perturbation detection on the key video frame to obtain a second detection score; obtaining a target detection score based on the first detection score and the second detection score, and acquiring key video frames whose target detection score is less than a preset score.

[0221] In some embodiments, the detection module (not shown in the figure) is further configured to perform latent space encoding and decoding on the key video frame to obtain a reconstructed video frame of the key video frame; determine a reconstruction error and probability divergence characterizing the category of the key video frame based on the reconstructed video frame and the key video frame; and obtain the first detection score based on the reconstruction error and probability divergence.

[0222] In some embodiments, the training module (not shown in the figure) is further configured to acquire a second sample dataset, the second sample dataset including multiple second video frames and a category label for each second video frame; wherein, the second video frame includes a target object; perform latent space encoding and decoding on the second video frame using an initial adversarial example detection model to obtain a reconstructed image corresponding to the second video frame; obtain the predicted category of the second video frame based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; obtain a second loss based on the predicted category and category label of the second video frame; and adjust the model parameters of the initial adversarial example detection model based on the second loss to obtain a trained adversarial example detection model.

[0223] In some embodiments, the detection module (not shown in the figure) is further configured to extract features from the key video frame to obtain a first feature map of the key video frame; perform a size transformation on the first feature map to obtain a second feature map; the second feature map includes the features in the first feature map; obtain a predicted value for the category of the key video frame based on the second feature map; and determine a second detection score based on the predicted value.

[0224] In some embodiments, the training module (not shown in the figure) is further configured to acquire a third sample dataset, which includes multiple third video frames and a category label for each third video frame; wherein the third video frame includes a target object; the third video frame is processed based on the initial fake sample detection model to obtain a predicted category; a third loss is obtained based on the predicted category and the category label; and the model parameters of the initial fake sample detection model are adjusted based on the third loss to obtain a trained fake sample detection model.

[0225] The descriptions of the above image processing apparatus embodiments are similar to those of the image processing method embodiments, and have similar beneficial effects. In some embodiments, the functions or modules included in the image processing apparatus provided in this application can be used to execute the methods described in the image processing method embodiments. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the image processing method embodiments of this application for understanding.

[0226] This application provides an electronic device, which can be an electronic device (e.g., a laptop, desktop computer, server cluster, etc.), including a processor and a memory. The memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory to implement the above-described method.

[0227] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method. The computer-readable storage medium can be transient or non-transient.

[0228] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions for executing all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Therefore, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0229] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed by the computer program, the processor performs some or all of the steps for implementing the above-described method.

[0230] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK).

[0231] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the image processing method embodiments of this application for understanding.

[0232] This application provides an electronic device, such as... Figure 15As shown, the hardware entity of the electronic device 1500 includes: a processor 1501, a communication interface 1502, and a memory 1503, wherein:

[0233] Processor 1501 typically controls the overall operation of electronic device 1500.

[0234] Communication interface 1502 enables electronic devices to communicate with other terminals or servers via a network.

[0235] The memory 1503 is configured to store instructions and applications executable by the processor 1501, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 1501 and various modules in the electronic device 1500. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 1501, the communication interface 1502, and the memory 1503 can be performed via bus 1504.

[0236] It should be understood that the phrase "an embodiment" or "one embodiment" mentioned throughout the specification indicates that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not indicate the order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The above embodiment numbers are for descriptive purposes only and do not indicate the superiority or inferiority of the embodiments.

[0237] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0238] In the several embodiments provided in this application, it should be understood that the disclosed apparatus, devices, and methods can be implemented in other ways. The apparatus and device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0239] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0241] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0242] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0243] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: Acquire at least one key video frame from the original video stream; the key video frame is a video frame containing the target object. Identify the target region where the target object is located in the key video frame; The target region is blurred to obtain a desensitized video frame; wherein, the image of the blurred region in the desensitized video frame is generated based on the image of the surrounding region of the target region; The method further includes: performing perturbation detection on the key video frames, and filtering the key video frames based on the detection results; the detection results include the target detection score of the key video frames; the perturbation detection on the key video frames and the filtering of the key video frames based on the detection results include: performing pixel perturbation detection on the key video frames to obtain a first detection score; performing forgery perturbation detection on the key video frames to obtain a second detection score; obtaining a target detection score based on the first detection score and the second detection score, and acquiring key video frames whose target detection score is less than a preset score.

2. The method according to claim 1, characterized in that, The process of blurring the target region to obtain a desensitized video frame includes: The key video frames are reconstructed to obtain reconstructed video frames; A desensitized block is obtained based on the surrounding image of the target region in the reconstructed video frame; The desensitized video frame is obtained based on the reconstructed video frame and the desensitized block.

3. The method according to claim 2, characterized in that, The process of obtaining the desensitized video frame is achieved through a trained blurring model, and the training process of the trained blurring model includes: Obtain a first sample dataset, which includes multiple first video frames and the original video frames corresponding to the first video frames; wherein, the pixel values ​​of the target region image in the first video frame are preset pixel values; The first video frame is reconstructed using an initial fuzzing model to obtain a reconstructed video frame. The reconstructed video frame is then processed to obtain a corresponding desensitized block. Based on the desensitized block and the reconstructed video frame, a corresponding predicted video frame is obtained. The reconstruction loss is then obtained based on the original video frame and the predicted video frame. The initial fuzzing model is used to extract features from the reconstructed video frames and the original video frames, resulting in a first feature map corresponding to the reconstructed video frames and a second feature map corresponding to the original video frames. The feature matching loss is then obtained based on the first and second feature maps. The reconstructed video frames are encoded using an initial fuzzing model to obtain the pixel distribution of the encoded data of the reconstructed video frames; the KL divergence loss is then obtained based on the pixel distribution and the normal distribution. The first loss is obtained based on the reconstruction loss, feature matching loss, and KL divergence loss; the model parameters of the initial fuzzy processing model are adjusted based on the first loss to obtain the trained fuzzy processing model.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining at least one key video frame from the original video stream includes: Each video frame in the original video stream is verified, and the video frames that meet the preset requirements are determined as the target video frames. Obtain the feature information of the target video frame in the original video stream; The feature information in the target video frame is analyzed, and the target video frame corresponding to the feature information that meets the preset requirements is determined as the key video frame.

5. The method according to claim 1, characterized in that, The step of performing pixel perturbation detection on the key video frames to obtain a first detection score includes: Latent space encoding and decoding are performed on the key video frames to obtain the reconstructed video frames of the key video frames; Based on the reconstructed video frames and the key video frames, determine the reconstruction error and probability divergence that characterize the category of the key video frames; The first detection score is obtained based on the reconstruction error and probability divergence.

6. The method according to claim 5, characterized in that, The reconstruction error and probability divergence for determining the category of the key video frame are achieved through an adversarial example detection model, the training process of which includes: Obtain a second sample dataset, which includes multiple second video frames and a category label for each second video frame; wherein, the second video frames include target objects; The second video frame is latent space encoded and decoded using the initial adversarial example detection model to obtain the reconstructed image corresponding to the second video frame; the predicted category of the second video frame is obtained based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; and the second loss is obtained based on the predicted category and category label of the second video frame. The model parameters of the initial adversarial example detection model are adjusted based on the second loss to obtain the trained adversarial example detection model.

7. The method according to claim 6, characterized in that, The step of performing forgery and perturbation detection on the key video frames to obtain a second detection score includes: Feature extraction is performed on the key video frame to obtain the first feature map of the key video frame; The first feature map is resized to obtain a second feature map; the second feature map includes the features in the first feature map. Based on the second feature map, a predicted value for the category of the key video frame is obtained; The second detection score is determined based on the predicted value.

8. The method according to claim 7, characterized in that, The process of obtaining the predicted value of the category for the key video frame is implemented through a trained forgery detection model, the training process of which includes: Obtain a third sample dataset, which includes multiple third video frames and a category label for each third video frame; wherein, the third video frames include target objects; The third video frame is processed based on the initial fake sample detection model to obtain the predicted category; the third loss is obtained based on the predicted category and category label. The model parameters of the initial fake sample detection model are adjusted based on the third loss to obtain the trained fake sample detection model.

9. An image processing apparatus, characterized in that, The device includes: An acquisition module is used to acquire at least one key video frame from the original video stream; the key video frame is a video frame with a target object. The identification module is used to identify the target area where the target object is located in the key video frame; The processing module is used to blur the target area to obtain a desensitized video frame; wherein, the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area; The acquisition module is further configured to perform perturbation detection on the key video frames and filter the key video frames based on the detection results; the detection results include the target detection score of the key video frames; the perturbation detection on the key video frames and the filtering of the key video frames based on the detection results include: performing pixel perturbation detection on the key video frames to obtain a first detection score; performing forgery perturbation detection on the key video frames to obtain a second detection score; obtaining a target detection score based on the first detection score and the second detection score, and acquiring key video frames whose target detection score is less than a preset score.

Images