Image processing method and device

Abstract

The invention provides an image processing method and device, and the method comprises the steps: obtaining at least one key video frame in an original video stream; the key video frame is a video frame with a target object; identifying a target area where a target object in the key video frame is located; performing fuzzy processing on the target area to obtain a desensitized video frame; wherein the image of the area subjected to fuzzy processing in the desensitized video frame is generated based on the image of the surrounding area of the target area. The target area is subjected to fuzzy processing through the image of the surrounding area of the target area so as to obtain the desensitized video frame with global features and local details similar to those of the key video frame corresponding to the desensitized video frame, and therefore the accuracy of the desensitized video frame is improved.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] In related technologies, desensitization processing usually involves mosaic processing, occlusion processing, obfuscation processing, and other methods. Mosaic processing is a simple and intuitive processing method that replaces pixels in an area to achieve a blurring effect; occlusion processing usually involves covering a specified area with a pattern of a specific color or texture to protect privacy; obfuscation processing methods usually involve transforming the color, brightness, or shape of a specified area so that features within the area are retained. The above methods all directly paste image blocks of the same color or texture, resulting in the loss of important features of the image and details of the feature map. Summary of the invention

[0003] In view of this, an embodiment of the present application provides an image processing method and device.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which includes: obtaining at least one key video frame in an original video stream; the key video frame is a video frame with a target object; identifying a target area where the target object in the key video frame is located; blurring the target area to obtain a desensitized video frame; wherein the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

[0006] In some embodiments, the key video frame is reconstructed to obtain a reconstructed video frame; a desensitized block is obtained based on a surrounding image of a target area in the reconstructed video frame; and the desensitized video frame is obtained based on the reconstructed video frame and the desensitized block.

[0007] According to the above-mentioned technical means, the key video frame is reconstructed to obtain a reconstructed video frame with less noise, and a desensitized block for the target area is generated based on the features of the surrounding image of the target area in the reconstructed video frame, so that the generated desensitized video frame and its corresponding key video frame have similar global features and local details, and the information entropy of the entire key video frame is retained, thereby improving the accuracy of the desensitized video frame.

[0008] In some embodiments, a first sample data set is obtained, wherein the first sample data set includes multiple first video frames and original video frames corresponding to the first video frames; wherein the pixel value of the image of the target area in the first video frame is a preset pixel value; the first video frame is reconstructed by an initial blur processing model to obtain a reconstructed video frame of the first video frame, and the reconstructed video frame is processed to obtain a corresponding desensitized block; a corresponding predicted video frame is obtained based on the desensitized block corresponding to the reconstructed video frame and the reconstructed video frame; a reconstruction loss is obtained based on the original video frame and the predicted video frame; feature extraction is performed on the reconstructed video frame and the original video frame by the initial blur processing model to obtain a first feature map corresponding to the reconstructed video frame and a second feature map corresponding to the original video frame; a feature matching loss is obtained based on the first feature map and the second feature map; the reconstructed video frame is encoded by the initial blur processing model to obtain a pixel distribution of the encoded data of the reconstructed video frame; a KL divergence loss is obtained based on the pixel distribution and the normal distribution; a first loss is obtained based on the reconstruction loss, the feature matching loss and the KL divergence loss; model parameters of the initial blur processing model are adjusted based on the first loss to obtain a trained blur processing model.

[0009] According to the above technical means, an initial blur processing model is trained through video frames with pixel values ​​of 0 in multiple target areas, and a first loss is obtained by calculating reconstruction loss, feature matching loss, and KL divergence loss. The model parameters of the initial blur processing model are adjusted by the first loss until the convergence condition is met, thereby obtaining a trained blur processing model. Based on the trained blur processing model, the key video frames are reconstructed to obtain reconstructed video frames, and the reconstructed video frames are blurred to obtain desensitized video frames having similar global features and local details to the key video frames, thereby improving the accuracy of the desensitized video frames.

[0010] In some embodiments, each video frame in the original video stream is checked, and a video frame that meets preset requirements is determined as a target video frame; feature information of the target video frame in the original video stream is obtained; the feature information in the target video frame is analyzed, and the target video frame corresponding to the feature information that meets the preset requirements is determined as the key video frame.

[0011] According to the above-mentioned technical means, the video frames in the original video stream are verified, and the video frames in the original video stream are screened based on the verification results to obtain target key frames. The feature information in the target key frames is extracted based on the principal component analysis and machine learning methods. At the same time, the target video frames are screened in combination with the time domain and spatial domain features of the feature information of the target video frames to obtain key video frames, thereby improving the screening efficiency of key video frames.

[0012] In some embodiments, the detection result includes a target detection score of the key video frame; the perturbation detection is performed on the key video frame, and the key video frame is screened based on the detection result, including: performing pixel perturbation detection on the key video frame to obtain a first detection score; performing forgery perturbation detection on the key video frame to obtain a second detection score; obtaining a target detection score based on the first detection score and the second detection score, and obtaining a key video frame whose target detection score is less than a preset score.

[0013] According to the above technical means, by detecting deep fake samples in key video frames, the key video frames are screened based on the detection scores to reduce the workload of subsequent blur processing and improve the efficiency of subsequent blur processing.

[0014] In some embodiments, the key video frame is latent-space encoded and decoded to obtain a reconstructed video frame of the key video frame; a reconstruction error and a probability divergence characterizing a category of the key video frame are determined based on the reconstructed video frame and the key video frame; and the first detection score is obtained based on the reconstruction error and the probability divergence.

[0015] According to the above-mentioned technical means, the key video frame is latent-space encoded and decoded to obtain its corresponding reconstructed video frame, so as to reduce the influence of the noise of the key video frame on subsequent detection, and the first detection score of the key video frame is determined based on the reconstruction error and probability divergence between the reconstructed video frame and the key video frame, and it is determined whether the key video frame is an adversarial sample, thereby increasing the workload of subsequent blur processing and improving the efficiency of blur processing.

[0016] In some embodiments, a second sample data set is obtained, wherein the second sample data set includes multiple second video frames and a category label for each second video frame; wherein the second video frame includes a target object; the second video frame is latent-space encoded and decoded through an initial adversarial sample detection model to obtain a reconstructed image corresponding to the second video frame; a predicted category of the second video frame is obtained based on a reconstruction loss and a probability divergence between the reconstructed image and the second video frame; a second loss is obtained based on the predicted category and the category label of the second video frame; and model parameters of the initial adversarial sample detection model are adjusted based on the second loss to obtain a trained adversarial sample detection model.

[0017] According to the above technical means, the key video frames are detected through the trained adversarial sample detection model to determine whether the key video frames are adversarial samples, so as to screen the key video frames to reduce the subsequent blurred processing data, thereby improving the accuracy of the blur processing.

[0018] In some embodiments, feature extraction is performed on the key video frame to obtain a first feature map of the key video frame; the first feature map is resized to obtain a second feature map; the second feature map includes features in the first feature map; a predicted value for the category of the key video frame is obtained based on the second feature map; and the second detection score is determined based on the predicted value.

[0019] According to the above technical means, the second feature map of the key video frame is analyzed to obtain a prediction value for the second feature map, and when the prediction value is greater than a preset threshold, the key video frame is characterized as a regional forged sample. Wherein, the prediction value and the preset threshold are values ​​between 0 and 1, and the prediction value greater than the preset threshold indicates that the key video frame is a regional forged sample, and the prediction value less than the preset threshold indicates that the key video frame is not a regional forged sample.

[0020] In some embodiments, a third sample data set is obtained, wherein the third sample data set includes multiple third video frames and a category label for each third video frame; wherein the third video frame includes a target object; the third video frame is processed based on an initial forged sample detection model to obtain a predicted category; a third loss is obtained based on the predicted category and the category label; and model parameters of the initial forged sample detection model are adjusted based on the third loss to obtain a trained forged sample detection model.

[0021] According to the above technical means, the key video frames are detected by the trained forged sample detection model to determine whether the key video frames are regional forged samples, so as to screen the key video frames to reduce the subsequent blurred processing data, thereby improving the accuracy of the blur processing.

[0022] In a second aspect, an embodiment of the present application provides an image processing device, which includes: an acquisition module for acquiring at least one key video frame in an original video stream; a key video frame is a video frame with a target object; an identification module for identifying a target area where the target object in the key video frame is located; a processing module for blurring the target area to obtain a desensitized video frame; wherein the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0025] The beneficial effects of the present application are as follows: by performing frame extraction processing on the original video stream to obtain a key video frame with a target object, the workload of subsequent blurring processing is reduced to reduce resource consumption; a desensitized block corresponding to the target area is generated through the features of the surrounding image of the target area, and the target area is blurred through the desensitized block to obtain a desensitized video frame with similar global features, local features and details to the corresponding key video frame, thereby improving the robustness, effectiveness and accuracy of the desensitized video frame.

[0026] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings (which are not necessarily drawn to scale), like reference numerals may describe similar components in different views. Like reference numerals with different letter suffixes may represent different examples of similar components. The accompanying drawings generally illustrate various embodiments discussed herein by way of example and not limitation.

[0028] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0029] Figure 2 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0034] Figure 7 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0035] Figure 8 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of an implementation flow of a key frame recognition method provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of a process for detecting deep fake samples provided in an embodiment of the present application;

[0038] Fig.11 A schematic diagram of the structure of an adversarial sample detection branch provided in an embodiment of the present application;

[0039] Fig.12 A schematic diagram of the structure of a regional forged sample detection branch provided in an embodiment of the present application;

[0040] Fig.13 A schematic diagram of the structure of a deep neural network provided in an embodiment of the present application;

[0041] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0042] Fig.15 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the following figure will further describe the present application in detail in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0044] The exemplary embodiments disclosed in the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the specific embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope disclosed in the present application to those skilled in the art.

[0045] In the following description, a large number of specific details are given to provide a more thorough understanding of the present application. However, it is obvious to those skilled in the art that the present application can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present application, some technical features known in the art are not described; that is, all features of the actual embodiments are not described here, and well-known functions and structures are not described in detail.

[0046] In the drawings, the sizes of layers, regions, elements and their relative sizes may be exaggerated for clarity. Like reference numerals denote like elements throughout.

[0047] The purpose of the terms used herein is only to describe specific embodiments and is not intended to be limiting of the present application. When used herein, the singular forms "one", "an" and "said / the" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "consisting of" and / or "comprising", when used in this specification, determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The technical and scientific terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] In the related art, desensitization processing algorithms usually use methods such as mosaic processing, occlusion processing, and obfuscation processing. Mosaic processing is a simple and intuitive processing method that achieves a blurring effect by replacing pixels in an area with pixel blocks with the same values ​​of surrounding pixels; occlusion processing usually takes the form of covering a specified area with a pattern of a specific color or texture to protect privacy; obfuscation processing methods usually take the form of color, brightness, or shape transformation of the specified area so that the features in the area are retained. The above methods all directly convert surrounding pixels into desensitized pixels in some way or paste image blocks of the same color or texture, which is bound to lose important features of the image and details of the feature map.

[0050] The present application provides an image processing method, which extracts frames from an original video stream to obtain a key video frame with a target object, thereby reducing the workload of subsequent blurring processing and reducing resource consumption; a desensitized block corresponding to the target area is generated through the features of the surrounding image of the target area, and the target area is blurred through the desensitized block to obtain a desensitized video frame with similar global features and local details to the corresponding key video frame, thereby improving the accuracy of the desensitized video frame.

[0051] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S103, combining Figure 1 The steps shown are explained.

[0052] Step S101: Obtain at least one key video frame in an original video stream.

[0053] The key video frame is a video frame having a target object.

[0054] In some embodiments, the original video stream may be video stream data of a real scene captured by a hardware camera.

[0055] In some embodiments, the target object may be a sensitive object such as a pedestrian's face or a license plate.

[0056] In some embodiments, after the original video stream data is collected based on the hardware camera, it is first necessary to perform frame extraction processing on the original video stream to obtain key video frames including the target object in the original video stream.

[0057] Step S102: Identify a target area where the target object in the key video frame is located.

[0058] In some embodiments, a target area where a target object in the key video frame is located can be detected by using a trained target detection model.

[0059] Among them, the training process of the trained target detection model includes: obtaining a sample data set, the sample data set includes multiple video frames and the annotation box of the target object in each video frame, inputting the multiple video frames into the target detection model to be trained to obtain a predicted detection box for the target object, obtaining a loss value based on the annotation box of the target object and the predicted annotation box, and adjusting the model parameters of the target detection model to be trained based on the loss value until the convergence condition is met to obtain the trained target detection model.

[0060] In some embodiments, the target detection model may include a convolutional neural network model (Visual Geometry Group Network, VGG), a real-time target detection model (You Only Look Once, YOLO), a single-stage target detection model (Single Shot MultiBox Detector, SSD), etc., which is not specifically limited in this application.

[0061] Step S103: blur the target area to obtain a desensitized video frame.

[0062] The image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

[0063] In some embodiments, the image of the surrounding area refers to the image of all areas outside the target area in the key video frame.

[0064] In some embodiments, an image of an area surrounding a target area in a key video frame is obtained, features of the image of the surrounding area are extracted, pixel blocks of the target area are generated based on the features of the image of the surrounding area, and the target area is blurred with the generated pixel blocks to obtain a desensitized video frame. It can be understood that the generated pixel blocks are pasted to the target area to obtain a desensitized video frame.

[0065] The feature of the image of the surrounding area may be pixel information around the area, including features such as color, texture, edge and shape.

[0066] In some embodiments, after obtaining the desensitized video frame, the method further includes obtaining a desensitized video stream based on the desensitized video frame and the original video stream.

[0067] In some embodiments, video frames that have not been blurred are obtained from the original video stream, and based on the timing of the original video stream, the desensitized video frames and the video frames that have been blurred are synthesized to obtain the desensitized video stream.

[0068] In an embodiment of the present application, frame extraction is performed on the original video stream to obtain a key video frame with a target object, thereby reducing the workload of subsequent blurring processing and reducing resource consumption; a desensitized block corresponding to the target area is generated by the features of the surrounding image of the target area, and the target area is blurred by the desensitized block to obtain a desensitized video frame having similar global features and local details to the corresponding key video frame, thereby improving the accuracy of the desensitized video frame.

[0069] Figure 2 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 1 , Figure 1 Step S103 in can be updated to step S201 to step S203, combining Figure 2 The steps shown are explained.

[0070] Step S201: reconstruct the key video frame to obtain a reconstructed video frame.

[0071] In some embodiments, based on image processing software, the pixels of the target area in the key video frame are set to 0, the key video frame with pixels of the target area image as 0 is latent space encoded to obtain a latent space expression of the key video frame, and the latent space expression is decoded to obtain a reconstructed video frame.

[0072] Step S202: obtaining a desensitized block based on the surrounding image of the target area in the reconstructed video frame.

[0073] In some embodiments, features of surrounding images of a target area in a reconstructed video frame are obtained, and a desensitization block for the target area is generated based on the features of the surrounding images of the target area, wherein the desensitization block has features similar to those of the surrounding images.

[0074] Exemplarily, a desensitization block having the same size as the target area is generated based on the color, texture, shape, edge and other features of the surrounding images of the target area.

[0075] Step S203: obtaining the desensitized video frame based on the reconstructed video frame and the desensitization block.

[0076] In some embodiments, the desensitized block obtained above is pasted to the target area of ​​the reconstructed video frame to obtain a desensitized video frame corresponding to the key video frame.

[0077] In an embodiment of the present application, the key video frame is latent space encoded and decoded to obtain a reconstructed video frame with less noise, and a desensitized block for the target area is generated based on the features of the surrounding image of the target area in the reconstructed video frame, so that the generated desensitized video frame and its corresponding key video frame have similar global features and local details, and the information entropy of the entire key video frame is retained, thereby improving the accuracy of the desensitized video frame.

[0078] In some embodiments, the process of obtaining the desensitized video frame is implemented by a trained blur processing model, and the training process of the trained blur processing model includes:

[0079] A first sample data set is obtained, wherein the first sample data set includes a plurality of first video frames and an original video frame corresponding to each first video frame; wherein the pixel value of the image of the target area in the first video frame is a preset pixel value.

[0080] In some embodiments, the target area of ​​the original video frame corresponding to the first video frame has a target object; wherein the target object in the original video frame can be a sensitive object, such as a face, a license plate, a location, etc., or it can be a natural scenery, such as mountains, rivers, buildings, etc.

[0081] The first video frame is reconstructed through an initial blur processing model to obtain a reconstructed video frame of the first video frame, and the reconstructed video frame is processed to obtain a corresponding desensitized block; based on the desensitized block corresponding to the reconstructed video frame and the reconstructed video frame, a corresponding predicted video frame is obtained; and a reconstruction loss is obtained based on the original video frame and the predicted video frame.

[0082] In some embodiments, the reconstruction loss is obtained based on the mean square error of the pixels of the reconstructed video frame and the mean square error of the pixels of the predicted video frame, please refer to formula (1).

[0083]

[0084] Among them, L recon is the reconstruction loss, x i To reconstruct the i-th pixel value in the video frame, is to predict the i-th pixel value in the video frame, and N is the total number of pixels in the reconstructed video frame.

[0085] The reconstructed video frame and the original video frame are subjected to feature extraction through an initial blur processing model to obtain a first feature map corresponding to the reconstructed video frame and a second feature map corresponding to the original video frame; and a feature matching loss is obtained based on the first feature map and the second feature map.

[0086] In some embodiments, a first feature root map of the reconstructed video frame and a second feature map of the predicted video frame are extracted based on the feature extraction network layer in the initial blur processing model, and a feature matching loss is obtained based on the mean square error between the first feature map and the second feature map, please refer to formula (2).

[0087]

[0088] Among them, L feture is the feature matching loss, represents the feature map in the reconstructed video frame extracted by the j-th feature extraction network layer, represents the feature map in the predicted video frame extracted by the j-th feature extraction network layer, and M represents the total number of feature maps.

[0089] The reconstructed video frame is encoded through an initial blur processing model to obtain pixel distribution of encoded data of the reconstructed video frame; and the KL divergence loss is obtained based on the pixel distribution and the normal distribution.

[0090] In some embodiments, a KL divergence loss is obtained based on the KL divergence of the encoded data of the reconstructed video frame and the normal distribution, please refer to formula (3).

[0091]

[0092] Among them, L KL is the KL divergence loss, represents the mean of the latent space representation of the reconstructed video frame, represents the variance of the latent space representation of the reconstructed video frame, N z Represents the dimensionality of the latent space representation of the reconstructed video frame.

[0093] A first loss is obtained based on the reconstruction loss, the feature matching loss and the KL divergence loss; and a model parameter of the initial blur processing model is adjusted based on the first loss to obtain a trained blur processing model.

[0094] In some embodiments, a first loss is obtained based on the reconstruction loss, feature matching loss, and KL divergence loss and their corresponding preset weights, please refer to formula (4).

[0095] L mask =λ recon *L recon +λ feature *L feature +λ KL *L KL Formula (4)

[0096] Among them, L mask is the first loss, λ recon is the preset weight of the reconstruction loss, λ feature is the preset weight of feature matching loss, λ KL The default weight for KL divergence loss.

[0097] A first loss is obtained based on the reconstruction loss, the feature matching loss and the KL divergence loss; and a model parameter of the initial blur processing model is adjusted based on the first loss to obtain a trained blur processing model.

[0098] In some embodiments, model parameters of the initial blur processing model are adjusted based on the first loss until the model parameters meet a convergence condition, and the trained blur processing model is output.

[0099] In an embodiment of the present application, an initial blur processing model is trained using video frames in which the pixel values ​​of multiple target areas are 0, and a first loss is obtained by calculating the reconstruction loss, the feature matching loss, and the KL divergence loss. The model parameters of the initial blur processing model are adjusted using the first loss until the convergence condition is met, thereby obtaining a trained blur processing model. The key video frames are reconstructed based on the trained blur processing model to obtain reconstructed video frames, and the reconstructed video frames are blurred to obtain desensitized video frames having similar global features and local details to the key video frames, thereby improving the accuracy of the desensitized video frames.

[0100] Figure 3 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The process may include steps S301 to S303. Figure 3 The steps shown are explained.

[0101] Step S301: Check each video frame in the original video stream, and determine the video frames that meet the preset requirements as target video frames.

[0102] In some embodiments, a Kaiser-Meyer-Olkin (KMO) and Bartlett's Test of Sphericity (BartLett) test are performed on each video frame in the original video stream, including: setting the KMO metric value to x1, the approximate chi-square value of the BartLett sphericity test value to x2, and the significance level to x3, performing KMO and BartLett tests on each video frame to obtain a corresponding test value for each video frame, and when the test value of the video frame satisfies x1>0.5, x2>3000, and x3<0.05, it indicates that the video frame meets preset requirements, thereby determining the video frame as a target video frame; wherein, the target video frame represents a video frame from which features can be extracted.

[0103] Step S302: Acquire feature information of a target video frame in the original video stream.

[0104] In some embodiments, the feature information of the target video frame includes a feature vector and a feature value of an image in the target video frame.

[0105] In some embodiments, the eigenvalues ​​and eigenvectors of the principal component factors in the target video frame are calculated based on a multivariate data analysis method.

[0106] Exemplarily, the multivariate data analysis method includes principal component analysis, which calculates the eigenvalues ​​and eigenvectors of the principal component factors in the target video frame based on the principal component analysis, including: representing the target video frame as a matrix, preprocessing the target video frame, such as grayscale, binarization, dilation, etc., to eliminate noise and interference information; standardizing the matrix so that its mean is 0 and its variance is 1; calculating the covariance matrix of the matrix to represent the correlation between each pixel in the target video frame; performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; wherein the eigenvalue represents the variance of the principal component factor, that is, the degree to which the principal component factor explains the variation of the data set; the eigenvector represents the direction of the principal component factor.

[0107] Step S303: Analyze the characteristic information in the target video frame, and determine the target video frame corresponding to the characteristic information that meets the preset requirements as the key video frame.

[0108] In some embodiments, the eigenvalues ​​and eigenvectors of the target video frames are analyzed, and the target video frames are screened based on the size of the target eigenvalues ​​and the direction of the eigenvectors to obtain the selected video frames. If the number of eigenvalues ​​and eigenvectors exceeds the quantity threshold, the eigenvalues ​​and eigenvectors are predicted based on the trained first image recognition model to obtain the predicted contribution rate, and the target video frames corresponding to the eigenvalues ​​and eigenvectors whose contribution rate reaches the preset threshold are determined as the selected video frames.

[0109] In some embodiments, the training process of the trained first image recognition model includes: obtaining a sample data set, the sample data includes feature vectors and feature values ​​of multiple video frames and a label of whether each video frame is a key video frame, inputting the feature vectors and feature values ​​of multiple video frames into the initial first image recognition model to obtain the predicted contribution rate of the feature vectors and feature values ​​that characterize whether the video frame is a key video frame, and adjusting the model parameters of the initial first image recognition model based on the predicted contribution rate and the loss value of the label to obtain the trained first image recognition model. The first image recognition model includes a fully connected layer and a Softmax layer.

[0110] In some embodiments, since the video stream is time-series continuous and the contents of adjacent frames are close and continuous, in order to better meet the requirements of capturing time-series information and modeling based on long-term dependencies while utilizing context information, a time-domain analysis is performed on the target video frame including the main component factors, including predicting the main component factors of the selected video frame based on the trained second image recognition model, so as to determine the selected video frame corresponding to the main component factors that meet the time-series characteristics of the original video stream as the key video frame. The key video frame obtained based on the selected video frame is evaluated for performance based on the determined key video frame to obtain the accuracy of the trained second image recognition model. The second image recognition model includes a fully connected layer and a Softmax layer.

[0111] In an embodiment of the present application, the video frames in the original video stream are verified, and the video frames in the original video stream are screened based on the verification results to obtain target key frames. The feature information in the target key frames is extracted based on the principal component analysis and machine learning method. At the same time, the target video frames are screened in combination with the time domain and spatial domain features of the feature information of the target video frames to obtain key video frames, thereby improving the screening efficiency of key video frames.

[0112] Figure 4 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 1 , Figure 1 Step S101 in the above method may further include step S401, combining Figure 4 The steps shown are explained.

[0113] Step S401: Perform disturbance detection on the key video frames, and filter the key video frames based on the detection results.

[0114] In some embodiments, deep fake sample detection is performed on key video frames to filter out key video frames of deep fake samples and retain original clean key video frames.

[0115] Among them, deep fake samples include adversarial samples and region-limited fake samples. Adversarial samples represent pixel-level perturbations that are indistinguishable to the human eye superimposed on clean samples. Region-limited fake samples usually refer to samples that exist in the real physical world in the form of patches. Adversarial samples and region-limited fake samples will mislead neural network-based target detection models to make incorrect predictions.

[0116] Figure 5 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 4 , the detection result includes the target detection score of the key video frame; Figure 4 Step S401 in can be updated to step S501 to step S503, combining Figure 5 The steps shown are explained.

[0117] Step S501: Perform pixel disturbance detection on the key video frame to obtain a first detection score.

[0118] In some embodiments, the key video frame is reconstructed to obtain a reconstructed video frame, and reconstruction losses of the reconstructed video frame and the key video frame are obtained based on pixels of the reconstructed video frame and pixels of the key video frame. When the reconstruction loss reaches a preset threshold, the key video frame is characterized as an adversarial sample.

[0119] In some embodiments, the probability divergence between the reconstructed video frame and the key video frame is obtained based on the Jensen-Shannon (JS) divergence or the information gain (Kullback-Leibler, KL) divergence of the key video frame and the JS divergence or KL divergence of the reconstructed video frame. When the probability divergence reaches a preset threshold, the key video frame is characterized as an adversarial sample.

[0120] In some embodiments, the first detection score is derived based on the reconstruction loss and the probability divergence.

[0121] Step S502: Perform forgery disturbance detection on the key video frame to obtain a second detection score.

[0122] In some embodiments, features are extracted from key video frames to obtain feature maps of the key video frames, and the feature maps of the key video frames are analyzed to obtain prediction values ​​of the key video frames. When the prediction value reaches a preset threshold, the key video frame is determined to be a region-limited forgery sample, and a second detection score is determined based on the prediction value.

[0123] Step S503: obtaining a target detection score based on the first detection score and the second detection score, and obtaining a key video frame in which the target detection score is less than a preset score.

[0124] In some embodiments, the target detection score is obtained based on the first detection score and the second detection score and the weight of the first detection score and the second detection score, please refer to formula (5).

[0125] F(x)=τ*F1(x)+(1-τ)*F2(x) Formula (5)

[0126] Among them, F(x) is the target detection score, F1(x) is the first detection score, F2(x) is the second detection score, τ is the weight of the first detection score, and (1-τ) is the weight of the second detection score.

[0127] Exemplarily, if the first detection score is F1(x)=[α1,β1] and the second detection score is F2(x)=[α2,β2], then the target detection score is F(x)=[τ·α1+(1-τ)·α2τβ1+(1-τ)·β2].

[0128] In an embodiment of the present application, deep fake samples are detected in key video frames, so that the key video frames are screened based on the target detection scores to reduce the workload of subsequent blurring processing and improve the efficiency of subsequent blurring processing.

[0129] Figure 6 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 5 , Figure 5 Step S501 in can be updated to step S601 to step S603, combining Figure 6 The steps shown are explained.

[0130] Step S601: perform latent space encoding and decoding on the key video frame to obtain a reconstructed video frame of the key video frame.

[0131] In some embodiments, the key video frame is latent-space encoded to obtain a latent-space expression of the key video frame, and the latent-space expression is decoded to obtain a reconstructed video frame of the key video frame.

[0132] Step S602: Determine a reconstruction error and a probability divergence that characterize a category of the key video frame based on the reconstructed video frame and the key video frame.

[0133] In some embodiments, reconstruction losses of the reconstructed video frame and the key video frame are obtained based on pixels of the reconstructed video frame and pixels of the key video frame. When the reconstruction loss reaches a preset threshold, the key video frame is characterized as an adversarial sample.

[0134] In some embodiments, the probability divergence between the reconstructed video frame and the key video frame is obtained based on the JS divergence or KL divergence of the key video frame and the JS divergence or KL divergence of the reconstructed video frame. When the probability divergence reaches a preset threshold, the key video frame is characterized as an adversarial sample.

[0135] Step S603: Obtain the first detection score based on the reconstruction error and the probability divergence.

[0136] In some embodiments, based on the weights corresponding to the reconstruction error and the probability divergence, a weighted sum of the reconstruction error and the probability divergence is used as the first detection score.

[0137] In an embodiment of the present application, the key video frame is latent-space encoded and decoded to obtain its corresponding reconstructed video frame to reduce the impact of the noise of the key video frame on subsequent detection, and the first detection score of the key video frame is determined based on the reconstruction error and probability divergence between the reconstructed video frame and the key video frame, and it is determined whether the key video frame is an adversarial sample, thereby increasing the workload of subsequent blur processing and improving the efficiency of blur processing.

[0138] In some embodiments, the above-mentioned determination of the reconstruction error and the probability divergence characterizing the category of the key video frame is implemented by an adversarial sample detection model, and the training process of the adversarial sample detection model includes:

[0139] A second sample data set is obtained, where the second sample data set includes a plurality of second video frames and a category label of each second video frame; wherein the second video frame includes a target object.

[0140] In some embodiments, the second video frame may be a historical video frame including a sensitive object, such as a video frame including a face, a license plate, etc., or a video frame including natural scenery or a building.

[0141] In some embodiments, the category label of the second video frame is a label representing whether the second video frame is an adversarial sample.

[0142] The second video frame is latently encoded and decoded through the initial adversarial sample detection model to obtain a reconstructed image corresponding to the second video frame; the predicted category of the second video frame is obtained based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; and the second loss is obtained based on the predicted category and category label of the second video frame.

[0143] In some embodiments, when the reconstruction loss is greater than a preset threshold, the second video frame is characterized as an adversarial sample; when the probability divergence is greater than a preset threshold, the second video frame is characterized as an adversarial sample.

[0144] The model parameters of the initial adversarial sample detection model are adjusted based on the second loss to obtain the trained adversarial sample detection model.

[0145] In some embodiments, a threshold for reconstruction loss and a threshold for probability divergence in the initial adversarial sample detection model are adjusted based on the second loss until a convergence condition is met, and a trained adversarial sample detection model is output.

[0146] In an embodiment of the present application, key video frames are detected through a trained adversarial sample detection model to determine whether the key video frames are adversarial samples, so as to screen the key video frames to reduce subsequent blurred processing data, thereby improving the accuracy of blur processing.

[0147] Figure 7 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application, which can be executed by a processor of an electronic device. Figure 5 , Figure 5 Step S502 in the above example can be updated to steps S701 to S704, which will be combined with Figure 7 The steps shown are explained.

[0148] Step S701: extract features from the key video frame to obtain a first feature map of the key video frame.

[0149] In some embodiments, the key video frames of the first size are preprocessed, including denoising, frame rate adjustment, color correction, etc., and features are extracted from the preprocessed key video frames to obtain a first feature map of the first size.

[0150] Step S702: resize the first feature map to obtain a second feature map; the second feature map includes features in the first feature map.

[0151] In some embodiments, the first feature map is resized to obtain a second feature map of a preset size.

[0152] Step S703: Obtain a prediction value for the category of the key video frame based on the second feature map.

[0153] In some embodiments, the second feature map of the key video frame is analyzed to obtain a prediction value for the second feature map, and when the prediction value is greater than a preset threshold, the key video frame is characterized as a regional forgery sample. Wherein, the prediction value and the preset threshold are values ​​between 0 and 1, the prediction value is greater than the preset threshold to characterize the key video frame as a regional forgery sample, and the prediction value is less than the preset threshold to characterize the key video frame is not a regional forgery sample.

[0154] Step S704: Determine the second detection score based on the predicted value.

[0155] In some embodiments, the predicted value is used as the second detection score.

[0156] In an embodiment of the present application, features are extracted from key video frames to obtain a first feature map of corresponding size, the first size map is resized to obtain a second feature map that meets the analysis requirements, and the second feature map is analyzed to obtain a predicted value for the category of the key video frame, and based on the predicted value, it is determined whether the key video frame is a regional forgery sample, so that the key video frame is screened to reduce the processing data of subsequent blurring, thereby improving the accuracy of blurring processing.

[0157] In some embodiments, the process of obtaining the predicted value for the category of the key video frame is implemented by a trained forged sample detection model, and the training process of the trained forged sample detection model includes:

[0158] A third sample data set is obtained, where the third sample data set includes a plurality of third video frames and a category label of each third video frame; wherein the third video frame includes a target object.

[0159] In some embodiments, the third video frame may be a historical video frame including a sensitive object, such as a video frame including a face, a license plate, etc., or a video frame including natural scenery or a building.

[0160] In some embodiments, the category label of the third video frame is a label indicating whether the third video frame is a region forgery sample.

[0161] The third video frame is processed based on the initial forged sample detection model to obtain a predicted category; and a third loss is obtained based on the predicted category and the category label.

[0162] In some embodiments, features are extracted from the third video frame based on an initial forged sample detection model to obtain a first feature map corresponding to the size of the third video frame, the first feature map is resized to obtain a second feature map of a size that can be analyzed by the initial forged sample detection model, and the second feature map is analyzed to obtain a prediction value characterizing whether the third video frame is a regional forged sample.

[0163] The model parameters of the initial forged sample detection model are adjusted based on the third loss to obtain a trained forged sample detection model.

[0164] In some embodiments, model parameters of the initial forged sample detection model are adjusted based on the third loss adjustment until a convergence condition is met, and the trained forged sample detection model is output.

[0165] In an embodiment of the present application, key video frames are detected by a trained forged sample detection model to determine whether the key video frames are regional forged samples, so as to screen the key video frames to reduce subsequent blurred processing data, thereby improving the accuracy of blur processing.

[0166] An exemplary application of the image processing method provided in an embodiment of the present application in a practical scenario is described below.

[0167] As users attach more importance to privacy protection, relevant policies and regulations require that real data (such as biometric features such as face and palm prints) collected by sensor hardware or camera equipment that may leak personal privacy be desensitized, and it is recommended to adhere to the desensitization principle during data processing and anonymize and de-identify sensitive data. In the field of smart cars, autonomous driving technology based on map modeling usually requires perception modeling of real roads. The real data collected by the data collection vehicle on the road through the camera is stored and processed, and the parameters of deep learning model training are tuned. Due to the limited data storage space and computing power of the vehicle, it does not support online training of road data collected in real time. The current common solution is to upload the data to the cloud and then provide the large model training data and optimization parameters on the server. Relevant policies and regulations require companies to legally protect public safety by desensitizing the face and license plate data in driving videos. At present, in order to meet the purpose of user privacy protection, smart cars usually choose to desensitize information such as face and license plate and then upload it offline to the cloud for large model training and parameter tuning in the cloud. The desensitization algorithm includes two steps: positioning the area to be desensitized and fuzzy processing. The desensitized area positioning is usually based on the detection model of text recognition or image recognition to locate the object to be desensitized in the video stream, and the corresponding coordinate rectangle is determined by locating the object. Blurring is usually mosaic processing or image replacement based on transfer learning methods.

[0168] In the related art, the video stream desensitization method directly performs regional detection and fuzzy processing on the data collected by the camera hardware. Among them, the regional detection method is to directly perform regional positioning on the collected video stream data, and the fuzzy processing method is based on the coordinate processing of the rectangular frame to be desensitized. If the rectangular frame positioning is abnormal, it will lead to user privacy leakage. At the same time, the mis-desensitization processing of other areas will mislead the abnormal classification of the neural network, and lose data value and resources. In real road conditions, deep fake samples have the ability to make the target detection model lose the target. With the continuous improvement and evolution of existing deep learning technology, adversarial samples and regional limited fake samples in real scenes will lead to incorrect prediction of the target area to be desensitized. Adversarial samples refer to the superposition of adversarial perturbations that are indistinguishable to the human eye on clean samples, thereby inducing neural network misclassification. The regional limited fake samples in the field of smart cars are to superimpose perturbation blocks visible to the human eye on license plates, traffic signs or traffic lights. Although visible to the human eye, they will produce abnormal processing results for autonomous driving vehicles. Due to the existence of adversarial samples and regional fake samples, the area to be desensitized is abnormal, which causes the desensitization algorithm to fail to desensitize or abnormally desensitize the original video stream, thereby causing privacy leakage.

[0169] In the embodiment of the present application, before desensitizing the video stream, the input video frame is subjected to deep forgery sample detection, including the identification of adversarial samples and region-limited forgery samples; the video stream desensitization method based on deep learning security protection can filter abnormal samples and improve the robustness of regional positioning, thereby improving the quality of desensitized video. At the same time, in order to reduce resource consumption, only the key frames of the video stream are processed instead of all video frames.

[0170] In the related technology, desensitization processing algorithms are usually blur processing, mosaic processing, occlusion processing, obfuscation processing and other methods. Blur processing uses the Gaussian discrete approximation processing, averaging and median value processing of the pixels in the specified area or the surrounding window to achieve smoothness and retain details, simple denoising and eliminate sudden noise. Mosaic processing is a simple and intuitive processing method that replaces the pixels in the area with pixel blocks with the same values ​​of the surrounding pixels to achieve a blurring effect; occlusion processing usually takes the form of covering the specified area with a pattern of a specific color or texture to protect privacy; obfuscation processing methods usually take the form of color, brightness or shape transformation of the specified area to retain the features in the area. However, these methods hardly consider the effect of the desensitized image on the subsequent reasoning effect based on the cloud-based large model. Usually, in the original image, if the surrounding pixels are directly converted into desensitized pixels in some way or the image blocks of the same color or texture are pasted, the important features of the image and the details of the feature map will inevitably be lost, thereby affecting the subsequent model detection accuracy and classification accuracy.

[0171] In order to ensure that the image with deletions and the real image have similar feature maps and details in the embodiment of the present application, a pre-trained VGG network is used to extract the features of the image, which includes multiple convolutional layers and pooling layers; the similarity between the repaired result and the original image is maintained by minimizing the feature space distance between them.

[0172] In an embodiment of the present application, deep learning security protection for key video frames can achieve robust performance in locating the area to be desensitized in video stream desensitization, while maintaining stable performance and reducing resource consumption.

[0173] Figure 8 The following is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The process includes steps S801 to S804. Figure 8 The steps shown are explained.

[0174] Step S801: For the original video stream data collected by the camera hardware, a key video frame is obtained by extracting the frame of the original video stream.

[0175] In some embodiments, if the input original video stream data is encoded data, the original video stream data needs to be decoded first.

[0176] In some embodiments, the original input video stream needs to be first subjected to frame extraction processing to reduce resource consumption. In the field of smart cars, the front-view camera and rear-view camera usually obtain a larger amount of data information and are more important, so fewer frames are extracted and the frame rate of the video stream is higher than other non-front and rear view cameras. At the same time, the priority of the surround view camera and the panoramic camera is adaptively adjusted based on the application scenario, and the video stream captured by the camera with a higher priority has a higher frame rate. Usually the frame rate of the video stream is maintained within the range of 40 frames.

[0177] Fig. 9 The following is a schematic diagram of a key frame recognition method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The method includes steps S901 to S904. Fig. 9 The steps shown are explained.

[0178] Step S901: verify the original video stream data.

[0179] In some embodiments, the original video stream data may be past historical video frames or video frames in a training set.

[0180] In some embodiments, it is necessary to consider performing KMO and BartLett checks on previous historical video frames or video frames in a training set, specifically including: setting the KMO metric value to x1, the approximate chi-square value of the BartLett spherical check value to x1 and the significance level to x3. In actual detection, in the embodiment of the present application, when x1>0.5, x2>3000, and x3<0.05, factor analysis can be performed on the data representing the video frame to identify the key frames of the original video stream data.

[0181] Step S902: extract feature information from the original video stream based on principal component analysis.

[0182] In some embodiments, the feature information includes feature vectors and feature values.

[0183] In some embodiments, the eigenvectors and eigenvalues ​​of the factors are calculated by principal component analysis on the historical video frames or the video stream data in the training set, wherein the data samples reduced by principal component analysis are selected based on the variance contribution rate of the factors to extract the main influencing factors and improve the accuracy of their weights.

[0184] Step S903: Screen the feature information based on the neural network fully connected layer and the Softmax layer to obtain the main component factors.

[0185] In some embodiments, if the number of calculated eigenvectors and eigenvalues ​​is large, in order to improve the key frame recognition speed, the eigenvectors and eigenvalues ​​obtained by principal component analysis are input into the cascade structure of the fully connected layer and the softmax layer for screening, including based on predetermined labels, converting to one-hot encoding and then performing cascade structure training based on the BP back propagation algorithm. For maturely trained models, effective and reasonable eigenvectors and eigenvalues ​​are screened out by comparing the predicted contribution rates to obtain effective and reasonable main component factors.

[0186] Step S904: Perform RNN time domain analysis on the video frames including the main component factors to obtain key video frames.

[0187] In some embodiments, since the video stream is time-continuous and the contents of adjacent frames are close and continuous, in order to better meet the needs of capturing time information and modeling based on long-term dependencies while utilizing contextual information, a time domain analysis is performed on video frames including main component factors, including: dividing historical video frames into a training set, a test set, and a validation set in a ratio of 7:2:1, extracting the time features including the main component factors of the key video frames in the training set, performing inference in the test set to obtain the key video frames, and finally evaluating the performance of the obtained key video frames based on the validation set.

[0188] Step S802: Detect key video frames to obtain target key video frames.

[0189] In some embodiments, deep fake sample detection is performed on key video frames, key video frames of deep fake samples are screened out, and original clean key video frames are retained as target key video frames.

[0190] In some embodiments, deep fake samples include adversarial samples and region-limited fake samples. Adversarial samples refer to pixel-level perturbations that are not distinguishable by the human eye and are superimposed on clean samples. Region-limited fake samples usually refer to patches that exist in the real physical world. Adversarial samples and region-limited fake samples will mislead the neural network-based target detection model to make incorrect predictions, that is, the target frame of the object to be desensitized will be abnormal, thereby losing the meaning of desensitization. In order to ensure that the positioning of the area to be desensitized is effective, the embodiment of the present application adopts a deep fake sample detection method that combines a dual-branch fusion of an adversarial sample detection branch and a region-limited fake sample detection branch to eliminate key video frames of adversarial samples and region-limited fake samples.

[0191] Fig.10 This is a flow chart of a deep fake sample detection method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The process includes steps S1001 to S1003. Fig.10 The steps shown are explained.

[0192] Step S1001: Detect key video frames based on the adversarial sample detection branch to obtain a first discrimination score.

[0193] In some embodiments, please refer to Fig.11 , Fig.11A structural schematic diagram of an adversarial sample detection branch provided in an embodiment of the present application includes an encoder 1101, a decoder 1102, and a deep neural network 1103, wherein the adversarial sample detection branch is trained through historical video frames to obtain a detection threshold of the adversarial sample detection branch; wherein the training process of the adversarial sample detection branch includes: inputting historical video frames as sample data into the anti-sample detection branch, and labeling information of whether the historical video frames are adversarial samples; mapping the historical video frames to a latent space through the encoder 1101 to obtain a latent space expression of the historical video frames, and then The decoder 1102 maps the latent space expression of the historical video frame to the original space to obtain a reconstructed video frame, obtains the reconstruction error between the historical video frame and the reconstructed video frame based on the deep neural network 1103, obtains the probability divergence between the historical video frame and the reconstructed video frame by calculating the JS divergence or KL divergence based on the deep neural network 1103, predicts the historical video frame based on the reconstruction error and the probability divergence to obtain a prediction result, adjusts the parameters of the deep neural network 1103 based on the prediction result and the loss value of the annotation information until the convergence condition is met, and outputs the trained adversarial sample detection branch. The parameters include the weight of determining the first discrimination score based on the reconstruction error and the probability divergence; and the threshold for judging whether the historical video frame is an adversarial sample based on the reconstruction error and the threshold for judging whether the historical video frame is an adversarial sample based on the probability divergence.

[0194] In some embodiments, the key video frame is input into a trained adversarial sample detection branch to obtain the reconstruction error and probability divergence of the key video frame, and whether the key video frame is an adversarial sample, and a first discrimination score is obtained based on the weights of the reconstruction error and probability divergence of the key video frame.

[0195] Step S1002: Detect the key video frame based on the regional forged sample detection branch to obtain a second identification score.

[0196] In some embodiments, the method of detecting the area to limit the forged sample focuses on checking whether there are disturbance blocks in multiple local areas of the image, so it can be implemented using convolutional neural networks (CNN). CNN focuses on the local image area through the local receptive field, and then integrates different local receptive fields in the deeper network layer to obtain the overall image information. This method integrates feature extraction, selection and classification into the entire model, improves the feature expression ability and enhances the distinguishability of features through end-to-end training, and the embodiment of the present application adopts a binary classifier based on transfer learning to realize the discrimination of forged samples. At the same time, in order to realize that samples with various resolution inputs can be adversarially identified with high precision, an adaptive input resolution sample processing module is integrated on the basis of the binary classifier based on transfer learning. The module adds an adaptive pooling layer and a convolution layer based on a small-size convolution kernel while keeping the feature extractor and feature classifier of the original network unchanged, thereby realizing the legitimacy discrimination of the samples to be tested with multiple input resolutions.

[0197] In some embodiments, the regional forged sample detection branch is trained based on historical video frames and annotation information that marks whether the historical video frames are regionally limited forged samples. The historical video frames are input into the regional forged sample detection branch to obtain a prediction value that characterizes whether the historical video frames are regionally limited forged samples. The parameters of the regional forged sample detection branch are adjusted based on the prediction value and the loss of annotation information until the convergence function is met, and the trained regional forged sample detection branch is output.

[0198] In some embodiments, the annotation information may be label information of 0 and 1, where 0 indicates that the historical video frame is a regional forged sample, and 1 indicates that the historical video frame is a clean sample, and the prediction value may be a value in the range of 0 to 1.

[0199] In some embodiments, please refer to Fig.12 , Fig.12 A structural schematic diagram of a regional forged sample detection branch provided for an embodiment of the present application includes a preprocessing module 1201 and a deep neural network module 1202, wherein the preprocessing module 1201 is used to preprocess the key video frames obtained above so that the processed key video frames meet the processing requirements of the deep neural network model 1202, and the processed key video frames are processed based on the deep neural network model 1202 to obtain weight parameters of the key video frames, and whether the key video frames are regional forged samples and a second discrimination score are determined based on the weight parameters.

[0200] In some embodiments, please refer to Fig.13 , Fig.13A structural schematic diagram of a deep neural network provided for an embodiment of the present application includes an input video frame 1301, a first feature map 1302, an adaptive pooling layer 1303, multiple small-sized convolution kernels 1304, a second feature map 1305, and a fully connected layer 1306. First, feature extraction is performed on input video frames 1301 of different sizes to obtain M first feature maps 1302 of different sizes. The M first feature maps 1302 of different sizes are input into the adaptive pooling layer 1303 and multiple small-sized convolution kernels 1304 for processing to obtain N second feature maps 1305 of the same size. The N second feature maps 1305 of the same size are input into the fully connected layer 1306 to obtain a predicted value of the input video frame corresponding to each second feature map 1305.

[0201] Step S1003: Determine a target key video frame based on the first identification score and the second identification score.

[0202] In some embodiments, a target recognition score is obtained based on the first recognition score and the second recognition score. Referring to the above formula (1), key video frames with target recognition scores greater than a preset score are eliminated to obtain target key video frames.

[0203] Step S803: Identify the desensitized area of ​​the target key video frame, and perform blur processing on the desensitized area to obtain a desensitized video frame.

[0204] In some embodiments, the desensitized area is an area including target features, such as facial information, license plate information, etc.

[0205] In some embodiments, if the surrounding pixels are directly converted into desensitized pixels in some way or the image blocks of the same color or texture are pasted, the important features of the image and the details of the feature map will inevitably be lost, thereby affecting the subsequent model detection accuracy and classification accuracy. Based on this, in order to ensure that the image with missing and the real image have similar feature maps and details, the embodiment of the present application intends to use a pre-trained deep neural network to extract the features of the image around the area to be desensitized, and the feature extraction network layer includes multiple convolutional layers and pooling layers. The embodiment of the present application hopes to maintain their similarity by minimizing the feature space distance between the repair result and the original image, wherein the embodiment of the present application uses reconstruction loss, feature matching loss and KL divergence combination as the loss function for training to generate desensitized blocks, which are pasted to the area to be desensitized to maximize the information entropy of the entire video frame for subsequent training and model reasoning.

[0206] In some embodiments, the key video frame is input into a trained target detection model, the desensitized area of ​​the target feature in the key video frame is identified, and the desensitized area is marked with a detection box to obtain a marked video frame.

[0207] In some embodiments, the preprocessed video frame is input into a trained blur processing model to obtain a desensitized video frame, wherein the blur processing model first sets the pixel value of the desensitized area to 0, generates a desensitized block in other areas based on the desensitized area with a pixel value of 0 and adds it to the desensitized area to obtain a desensitized video frame, and the desensitized video frame has similar global features and local details to the video frame in the original video stream.

[0208] In some embodiments, the blur processing model is composed of a variational autoencoder and a deep neural network, and the training process of the above-trained blur processing model includes: obtaining a sample data set, the sample data set includes a plurality of historical video frames, and the desensitized area where the target area in each historical video frame is located is marked by a detection box; the plurality of historical video frames are input into the variational autoencoder to obtain a plurality of first video frames, and the variational autoencoder sets the pixel value of the desensitized area of ​​each first video frame to 0; the plurality of first video frames are input into the blur processing model to be trained to fill the desensitized area with pixels from the surrounding area of ​​the desensitized area to obtain a second video frame; the reconstruction loss of the first video frame and the second video frame is obtained; the first feature map of the historical video frame and the second feature map of the first video frame are extracted to obtain the feature matching loss of the first feature map and the second feature map, the historical video frame is encoded to obtain the latent space expression of the historical video frame, the pixel distribution of the latent space expression is obtained, and the KL divergence loss is obtained based on the pixel distribution and the normal distribution; the target loss is obtained based on the reconstruction loss, the feature matching loss, the KL divergence loss and the corresponding weights, and the parameters of the blur processing model to be trained are adjusted based on the target loss to obtain the trained blur processing model. The sample data set may also use video frames that are irrelevant to the target features, such as natural scenery pictures.

[0209] In some embodiments, for the reconstruction loss, the mean square error is used as the reconstruction loss to measure the difference between the restoration result and the original image at the pixel level, referring to the above formula (1).

[0210] In some embodiments, for feature matching loss, the features of the restoration result and the original image are extracted based on the VGG network as a feature extraction network, and the differences between them are compared. A feature map of a certain layer can be selected to calculate the loss, or feature maps of multiple layers can be weighted combined. The difference between features is measured using mean square error or other distance metrics, and reference can be made to the above formula (2).

[0211] For the KL divergence loss, the pixel distribution expressed in the latent space of the input video frame is constrained to be close to the normal distribution, that is, the unit Gaussian distribution, and the KL divergence loss is obtained, referring to the above formula (3).

[0212] In some embodiments, referring to the above formula (4), based on the reconstruction loss, feature matching loss, KL divergence loss and the corresponding preset weights, a total loss is obtained to train the blur processing model.

[0213] Step S804: Obtain desensitized video stream data based on the desensitized target key video frame and the original video stream data.

[0214] In some embodiments, the desensitized video frames are combined with the original video frames and packaged into a complete desensitized video stream file in combination with the timing.

[0215] In an embodiment of the present application, frame extraction is performed on the original video stream to obtain a key video frame with a target object, thereby reducing the workload of subsequent blurring processing and reducing resource consumption; a desensitized block corresponding to the target area is generated by the features of the surrounding image of the target area, and the target area is blurred by the desensitized block to obtain a desensitized video frame having similar global features and local details to the corresponding key video frame, thereby improving the accuracy of the desensitized video frame.

[0216] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.14 As shown, the image processing device 1400 includes: an acquisition module 1401, an identification module 1402, and a processing module 1403, wherein: the acquisition module 1401 is used to acquire at least one key video frame in the original video stream; the key video frame is a video frame with a target object; the identification module 1402 is used to identify the target area where the target object in the key video frame is located; the processing module 1403 is used to blur the target area to obtain a desensitized video frame; wherein the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

[0217] In some embodiments, the processing module 1403 is also used to reconstruct the key video frame to obtain a reconstructed video frame; obtain a desensitized block based on a surrounding image of the target area in the reconstructed video frame; and obtain the desensitized video frame based on the reconstructed video frame and the desensitized block.

[0218] In some embodiments, the image processing device 1400 further includes a training module (not shown in the figure), which is used to obtain a first sample data set, wherein the first sample data set includes multiple first video frames and original video frames corresponding to the first video frames; wherein the pixel value of the image of the target area in the first video frame is a preset pixel value; the first video frame is reconstructed by an initial blur processing model to obtain a reconstructed video frame of the first video frame, and the reconstructed video frame is processed to obtain a corresponding desensitized block; a corresponding predicted video frame is obtained based on the desensitized block corresponding to the reconstructed video frame and the reconstructed video frame; a reconstruction loss is obtained based on the original video frame and the predicted video frame; The reconstructed video frame and the original video frame are subjected to feature extraction through the initial blur processing model to obtain a first feature map corresponding to the reconstructed video frame and a second feature map corresponding to the original video frame; a feature matching loss is obtained based on the first feature map and the second feature map; the reconstructed video frame is encoded through the initial blur processing model to obtain a pixel distribution of the encoded data of the reconstructed video frame; a KL divergence loss is obtained based on the pixel distribution and the normal distribution; a first loss is obtained based on the reconstruction loss, the feature matching loss and the KL divergence loss; and model parameters of the initial blur processing model are adjusted based on the first loss to obtain a trained blur processing model.

[0219] The acquisition module 1401 is also used to verify each video frame in the original video stream, and determine the video frame that meets the preset requirements as the target video frame; obtain the feature information of the target video frame in the original video stream; analyze the feature information in the target video frame, and determine the target video frame corresponding to the feature information that meets the preset requirements as the key video frame.

[0220] In some embodiments, the image processing device 1400 also includes a detection module (not shown in the figure), which is used for the detection results including the target detection score of the key video frame; the perturbation detection on the key video frame, and the screening of the key video frame based on the detection result, including: performing pixel perturbation detection on the key video frame to obtain a first detection score; performing forgery perturbation detection on the key video frame to obtain a second detection score; obtaining a target detection score based on the first detection score and the second detection score, and obtaining a key video frame whose target detection score is less than a preset score.

[0221] In some embodiments, the detection module (not shown in the figure) is also used to perform latent space encoding and decoding on the key video frame to obtain a reconstructed video frame of the key video frame; determine a reconstruction error and a probability divergence that characterize the category of the key video frame based on the reconstructed video frame and the key video frame; and obtain the first detection score based on the reconstruction error and the probability divergence.

[0222] In some embodiments, the training module (not shown in the figure) is also used to obtain a second sample data set, which includes multiple second video frames and a category label for each second video frame; wherein the second video frame includes a target object; the second video frame is latently encoded and decoded through the initial adversarial sample detection model to obtain a reconstructed image corresponding to the second video frame; the predicted category of the second video frame is obtained based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; the second loss is obtained based on the predicted category and category label of the second video frame; the model parameters of the initial adversarial sample detection model are adjusted based on the second loss to obtain a trained adversarial sample detection model.

[0223] In some embodiments, the detection module (not shown in the figure) is also used to extract features from the key video frame to obtain a first feature map of the key video frame; resize the first feature map to obtain a second feature map; the second feature map includes the features in the first feature map; obtain a predicted value for the category of the key video frame based on the second feature map; and determine the second detection score based on the predicted value.

[0224] In some embodiments, the training module (not shown in the figure) is also used to obtain a third sample data set, wherein the third sample data set includes multiple third video frames and a category label for each third video frame; wherein the third video frame includes a target object; the third video frame is processed based on the initial forged sample detection model to obtain a predicted category; a third loss is obtained based on the predicted category and the category label; and the model parameters of the initial forged sample detection model are adjusted based on the third loss to obtain a trained forged sample detection model.

[0225] The description of the above image processing device embodiment is similar to the description of the image processing method embodiment, and has similar beneficial effects as the same method embodiment. In some embodiments, the functions or modules included in the image processing device provided in the embodiment of the present application can be used to execute the method described in the image processing method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the image processing method embodiment of the present application for understanding.

[0226] An electronic device provided in an embodiment of the present application may be an electronic device (such as a laptop computer, a desktop computer, a server cluster, etc.) when implemented, including a processor and a memory, wherein the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory to implement the above method.

[0227] The embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above method is implemented. The computer-readable storage medium may be transient or non-transient.

[0228] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including several instructions for executing all or part of the above methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0229] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in the computer program, the processor executes some or all of the steps in the above method.

[0230] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK).

[0231] It should be pointed out here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the image processing method embodiment of this application for understanding.

[0232] An electronic device is provided in an embodiment of the present application, such as Fig.15As shown, the hardware entity of the electronic device 1500 includes: a processor 1501, a communication interface 1502 and a memory 1503, wherein:

[0233] The processor 1501 generally controls the overall operation of the electronic device 1500 .

[0234] The communication interface 1502 may enable the electronic device to communicate with other terminals or servers via a network.

[0235] The memory 1503 is configured to store instructions and applications executable by the processor 1501, and can also cache data to be processed or processed by the processor 1501 and each module in the electronic device 1500 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data transmission can be performed between the processor 1501, the communication interface 1502, and the memory 1503 through the bus 1504.

[0236] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification characterizes that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not characterize the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are only for description and do not characterize the advantages and disadvantages of the embodiments.

[0237] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0238] In several embodiments provided in the present application, it should be understood that the disclosed devices, equipment and methods can be implemented in other ways. The above-described device and equipment embodiments are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0239] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0240] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0241] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0242] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0243] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire at least one key video frame in the original video stream; the key video frame is a video frame having a target object; Identifying a target area where a target object in the key video frame is located; The target area is blurred to obtain a desensitized video frame; wherein the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

2. The method according to claim 1, characterized in that The blurring the target area to obtain a desensitized video frame includes: Reconstructing the key video frame to obtain a reconstructed video frame; Obtaining a desensitized block based on a surrounding image of the target area in the reconstructed video frame; The desensitized video frame is obtained based on the reconstructed video frame and the desensitization block.

3. The method according to claim 2, characterized in that The process of obtaining the desensitized video frame is implemented by a trained fuzzy processing model, and the training process of the trained fuzzy processing model includes: Acquire a first sample data set, wherein the first sample data set includes a plurality of first video frames and original video frames corresponding to the first video frames; wherein the pixel value of the image of the target area in the first video frame is a preset pixel value; Reconstruct the first video frame through the initial blur processing model to obtain a reconstructed video frame of the first video frame, and process the reconstructed video frame to obtain a corresponding desensitized block; obtain a corresponding predicted video frame based on the desensitized block corresponding to the reconstructed video frame and the reconstructed video frame; obtain a reconstruction loss based on the original video frame and the predicted video frame; Extract features of the reconstructed video frame and the original video frame through the initial blur processing model to obtain a first feature map corresponding to the reconstructed video frame and a second feature map corresponding to the original video frame; obtain feature matching loss based on the first feature map and the second feature map; The reconstructed video frame is encoded by the initial blur processing model to obtain the pixel distribution of the encoded data of the reconstructed video frame; the KL divergence loss is obtained based on the pixel distribution and the normal distribution; A first loss is obtained based on the reconstruction loss, the feature matching loss and the KL divergence loss; and a model parameter of the initial blur processing model is adjusted based on the first loss to obtain a trained blur processing model.

4. The method according to any one of claims 1 to 3, characterized in that: The obtaining of at least one key video frame in the original video stream comprises: Verifying each video frame in the original video stream, and determining a video frame that meets a preset requirement as a target video frame; Acquire feature information of a target video frame in the original video stream; The characteristic information in the target video frame is analyzed, and the target video frame corresponding to the characteristic information that meets preset requirements is determined as the key video frame.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Performing disturbance detection on the key video frames, and screening the key video frames based on the detection results; The detection result includes a target detection score of the key video frame; the perturbation detection on the key video frame and screening the key video frame based on the detection result includes: Performing pixel disturbance detection on the key video frame to obtain a first detection score; Performing forgery disturbance detection on the key video frame to obtain a second detection score; A target detection score is obtained based on the first detection score and the second detection score, and a key video frame in which the target detection score is less than a preset score is obtained.

6. The method according to claim 5, characterized in that The performing pixel disturbance detection on the key video frame to obtain a first detection score includes: Performing latent space encoding and decoding on the key video frame to obtain a reconstructed video frame of the key video frame; determining a reconstruction error and a probability divergence characterizing a class of the key video frame based on the reconstructed video frame and the key video frame; The first detection score is obtained based on the reconstruction error and the probability divergence.

7. The method according to claim 6, characterized in that The determining of the reconstruction error and the probability divergence characterizing the category of the key video frame is implemented by an adversarial sample detection model, and the training process of the adversarial sample detection model includes: Acquire a second sample data set, wherein the second sample data set includes a plurality of second video frames and a category label of each second video frame; wherein the second video frame includes a target object; The second video frame is encoded and decoded in the latent space by the initial adversarial sample detection model to obtain a reconstructed image corresponding to the second video frame; a predicted category of the second video frame is obtained based on the reconstruction loss and probability divergence between the reconstructed image and the second video frame; a second loss is obtained based on the predicted category and category label of the second video frame; The model parameters of the initial adversarial sample detection model are adjusted based on the second loss to obtain the trained adversarial sample detection model.

8. The method according to claim 5, characterized in that The performing forgery disturbance detection on the key video frame to obtain a second detection score includes: Performing feature extraction on the key video frame to obtain a first feature map of the key video frame; Performing a size transformation on the first feature map to obtain a second feature map; the second feature map includes features in the first feature map; Obtaining a prediction value for the category of the key video frame based on the second feature map; The second detection score is determined based on the predicted value.

9. The method according to claim 8, characterized in that The process of obtaining the predicted value for the category of the key video frame is implemented by a trained forged sample detection model, and the training process of the trained forged sample detection model includes: Acquire a third sample data set, wherein the third sample data set includes a plurality of third video frames and a category label of each third video frame; wherein the third video frame includes a target object; Processing the third video frame based on the initial forged sample detection model to obtain a predicted category; obtaining a third loss based on the predicted category and the category label; The model parameters of the initial forged sample detection model are adjusted based on the third loss to obtain a trained forged sample detection model.

10. An image processing device, characterized in that: The device comprises: An acquisition module, used to acquire at least one key video frame in the original video stream; the key video frame is a video frame having a target object; An identification module, used to identify a target area where a target object in the key video frame is located; A processing module is used to blur the target area to obtain a desensitized video frame; wherein the image of the blurred area in the desensitized video frame is generated based on the image of the surrounding area of ​​the target area.

Images