Privacy protection method and device for monitoring video images
By building a style migration model and discriminator network, generating and adjusting the loop link loss function, the problem of low privacy protection of traditional surveillance video images is solved, and the desensitization of sensitive information is achieved while retaining texture and color styles is realized. , improves the application authenticity of surveillance video images.
Patent Information
- Application Number
- CN202111110376.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-18
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-09-18
AI Technical Summary
Traditional privacy protection methods for surveillance video images lead to a reduction in the authenticity of video images, which makes it impossible to effectively protect private information and retain scene information.
By building a style migration model and discriminator network, generating and adjusting the circular link loss function, a stable monitoring privacy protection model is formed, converting the monitoring original image into a privacy protection image, retaining texture and color styles, while desensiting sensitive information.
It improves the authenticity of surveillance video images in the application, effectively protects private information, retains scene content, and enhances the authenticity of privacy protection.
Smart Images

Figure CN113793258B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a privacy protection method and device for monitoring video images. Background Art
[0002] With the development of the science and technology industry, surveillance cameras are everywhere in the streets and alleys. The surveillance video images captured by surveillance cameras play an important role in many tasks such as urban traffic management, vehicle identification, parking fee management, violation handling, and pedestrian re-identification. In different application scenarios, or when different people view the same video image, it is necessary to protect the privacy of the private area information outside the target area required by the person in the frame image. The privacy area information can include private information such as pedestrians, vehicles, and license plates in the surveillance video image.
[0003] However, in order to protect the privacy information of users, the traditional privacy protection method of surveillance video images only transmits the image of the moving target with a completely black or white background, rather than the image captured by the camera. Since the traditional privacy protection method fills the background with completely black or white, only the image of the moving target is retained, which reduces the authenticity of the surveillance video image when it is applied. Summary of the invention
[0004] The purpose of the present invention is to solve the technical problem that the authenticity of the traditional privacy protection method of monitoring video images is low. To achieve the above purpose, the present invention provides a privacy protection method and device for monitoring video images.
[0005] The present invention provides a privacy protection method for monitoring video images, comprising:
[0006] Obtain a style image set and monitor the original image set;
[0007] Generate a first generated image and a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set, wherein the first generated image is obtained by inputting the monitored original image and the style image into the original style transfer model, and the second generated image is obtained by inputting the style image and the monitored original image into the style original transfer model;
[0008] A first loop link is formed according to the monitored original image, the style image, the original style transfer model, the first generated image, the style original transfer model, and the third generated image, and a second loop link is formed according to the style image, the monitored original image, the style original transfer model, the second generated image, the original style transfer model, and the fourth generated image, wherein the third generated image is obtained by inputting the first generated image into the style original transfer model, and the fourth generated image is obtained by inputting the second generated image into the original style transfer model;
[0009] Adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model to form a stable surveillance privacy protection model;
[0010] The original surveillance image is converted into a privacy-preserving image according to the stable surveillance privacy-preserving model.
[0011] In one embodiment, generating a first generated image according to the monitored original image in the monitored original image set and the style image in the style image set includes:
[0012] Inputting the monitored original image and the style image into an original style generator network, and outputting the first generated image;
[0013] Inputting the style image and the first generated image into an original style discriminator network to discriminate the style image and the first generated image;
[0014] The original style generator network and the original style discriminator network form the original style transfer model.
[0015] In one embodiment, generating a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set includes:
[0016] Inputting the style image and the monitoring original image into a style original generator network, and outputting the second generated image;
[0017] Inputting the monitoring original image and the second generated image into a style original discriminator network to discriminate the monitoring original image and the second generated image;
[0018] The style primitive generator network and the style primitive discriminator network form the style primitive migration model.
[0019] In one embodiment, a first loop link is formed according to the monitoring original image, the first generated image and the third generated image, and the third generated image is obtained by inputting the first generated image into the style original generator network.
[0020] In one embodiment, a second loop link is formed according to the style image, the second generated image and the fourth generated image, and the fourth generated image is obtained by inputting the second generated image into the original style generator network.
[0021] In one embodiment, adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model to form a stable surveillance privacy protection model includes:
[0022] According to the loss function of the monitoring privacy protection model, adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial monitoring privacy protection model;
[0023] The loss function of the monitoring privacy protection model is:
[0024]
[0025] Where X represents the original monitoring image, Y represents the style image, G represents the original style generator network, and D Y represents the original style discriminator network, F represents the original style generator network, D X represents the original style discriminator network, L GAN (G,D Y ,X,Y) represents the loss function of the original style transfer model, L GAN (F,D X ,Y,X) represents the loss function of the original style transfer model, λ is the weight parameter, represents the loss function of the first loop link, represents a loss function of the second loop link;
[0026] Adjusting the loss function of the monitoring privacy protection model to a target equilibrium point to form a stable monitoring privacy protection model;
[0027] The target balance point is:
[0028]
[0029] Among them, G* represents the weight of the original style discriminator network, F * represents the weight of the original generator network of the style, and the min-max function represents finding a discriminator network so that the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized, and the difference between the feature distribution of the input image and the generated distribution of the generator network is minimized.
[0030] In one embodiment, the privacy protection method of the surveillance video image further includes:
[0031] The target image area of the privacy protection image is replaced with the target image area of the original monitoring image to obtain a privacy-protected monitoring video image.
[0032] In one embodiment, the present invention provides a privacy protection device for monitoring video images, comprising:
[0033] An acquisition module is used to acquire a style image set and monitor an original image set;
[0034] An image generation module, configured to generate a first generated image and a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set, wherein the first generated image is obtained by inputting the monitored original image and the style image into the original style transfer model, and the second generated image is obtained by inputting the style image and the monitored original image into the style original transfer model;
[0035] a loop link module, configured to form a first loop link according to the monitoring original image, the first generated image, and a third generated image, and to form a second loop link according to the style image, the second generated image, and a fourth generated image, wherein the third generated image is obtained by inputting the first generated image into the style original migration model, and the fourth generated image is obtained by inputting the second generated image into the original style migration model;
[0036] A model optimization module, used to adjust the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model to form a stable surveillance privacy protection model;
[0037] The privacy protection image generation module is used to convert the monitoring original image into a privacy protection image according to the stable monitoring privacy protection model.
[0038] In one embodiment, the image generation module includes an original style transfer module. The original style transfer module is used to input the monitored original image and the style image into an original style generator network, output the first generated image, and input the style image and the first generated image into an original style discriminator network to discriminate the style image and the first generated image.
[0039] In one embodiment, the image generation module further includes a style primitive transfer module. The style primitive transfer module is used to input the style image and the monitoring primitive image into a style primitive generator network, output the second generated image, and input the monitoring primitive image and the second generated image into a style primitive discriminator network to discriminate the monitoring primitive image and the second generated image.
[0040] In one embodiment, the third generated image in the recurrent link module is obtained by inputting the first generated image into the style original generator network.
[0041] In one embodiment, in the recurrent link module, the fourth generated image is obtained by inputting the second generated image into the original style generator network.
[0042] In one embodiment, the model optimization module includes:
[0043] A parameter adjustment module, configured to adjust the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model according to the loss function of the surveillance privacy protection model;
[0044] Among them, the loss function of the monitoring privacy protection model is:
[0045]
[0046] X represents the original monitoring image, Y represents the style image, G represents the original style generator network, and D Y represents the original style discriminator network, F represents the original style generator network, D X represents the original style discriminator network, L GAN (G,D Y ,X,Y) represents the loss function of the original style transfer model, L GAN (F,D X ,Y,X) represents the loss function of the original style transfer model, λ is the weight parameter, represents the loss function of the first loop link, represents a loss function of the second loop link;
[0047] A model generation module adjusts the loss function of the monitoring privacy protection model to a target equilibrium point to form a stable monitoring privacy protection model;
[0048] The target balance point is:
[0049]
[0050] Among them, G * represents the weight of the original style discriminator network, F * represents the weight of the original generator network of the style, and the min-max function represents finding a discriminator network so that the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized, and the difference between the feature distribution of the input image and the generated distribution of the generator network is minimized.
[0051] In one embodiment, the apparatus further comprises:
[0052] The monitoring video image generation module is used to replace the target image area of the privacy protection image with the target image area of the monitoring original image to obtain the monitoring video image after privacy protection.
[0053] In the privacy protection method of the above-mentioned surveillance video image, the stylized image set and the surveillance original image set are constructed, and the model optimization and iteration are performed according to the original style transfer model, the style original transfer model, the first loop link, the second loop link and the corresponding loss function to obtain the surveillance privacy protection model. Thus, the surveillance original image that needs to be style-transferred is sent to the surveillance privacy protection model to obtain the image after style transfer, that is, the privacy-protected image. The privacy-protected image is similar to the style image in terms of texture, color and other stylization, and sensitive information such as character features and license plate numbers is desensitized, but the content and scene information of the surveillance original image are still retained, thereby improving the authenticity of the surveillance video image when it is used. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a flowchart of the privacy protection method of monitoring video images provided by the present invention;
[0055] Figure 2 is a schematic diagram of a first loop link and a second loop link provided by the present invention;
[0056] Figure 3 It is a flowchart of the privacy protection method of monitoring video images provided by the present invention;
[0057] Figure 4The figure is a schematic diagram of the principle structure of the privacy protection device for monitoring video images provided by the present invention. DETAILED DESCRIPTION
[0058] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.
[0059] See also Figure 1 The present invention provides a privacy protection method for monitoring video images, comprising:
[0060] S10, obtaining a style image set and a monitoring original image set;
[0061] S20, generating a first generated image and a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set, wherein the first generated image is obtained by inputting the monitored original image and the style image into an original style transfer model, and the second generated image is obtained by inputting the style image and the monitored original image into an original style transfer model;
[0062] S30, forming a first loop link according to the monitored original image, the style image, the original style transfer model, the first generated image, the style original transfer model, and the third generated image, and forming a second loop link according to the style image, the monitored original image, the style original transfer model, the second generated image, the original style transfer model, and the fourth generated image, wherein the third generated image is obtained by inputting the first generated image into the style original transfer model, and the fourth generated image is obtained by inputting the second generated image into the original style transfer model;
[0063] S40, adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model to form a stable surveillance privacy protection model;
[0064] S50: Convert the original surveillance image into a privacy-preserving image according to the stable surveillance privacy-preserving model.
[0065] In S10, the style image set includes a plurality of style images. The style images may be collected through network collection, interception of video frames, etc. The style image set may be an abstract style such as cartoon style, illustration style, oil painting style, etc. The abstract style may be a cartoon style, illustration style, oil painting style, etc.
[0066] In S20, the monitoring original image and the style image are input into the original style transfer model, and the first generated image is output. The original style transfer model is used to map the monitoring original image to the style image and distinguish the style image from the first generated image. The number of the monitoring original images input into the original style transfer model may be one, two, or three, etc. The number of the style images input into the original style transfer model may be one, two, or three, etc. The first generated image is an image obtained after the monitoring original image is mapped to the style image. The original style transfer model can be used to determine whether the generated first generated image is real by distinguishing the style image from the first generated image. Whether the first generated image is real can be reflected by the loss function of the original style transfer model.
[0067] The style image and the monitoring original image are input into the style original transfer model, and the second generated image is output. The style original transfer model is used to map the style image to the monitoring original image and distinguish the monitoring original image from the second generated image. The number of the style images input into the style original transfer model can be one, two, or three, etc. The number of the monitoring original images input into the style original transfer model can be one, two, or three, etc. The second generated image is an image obtained after the style image is mapped to the monitoring original image. The style original transfer model can be used to determine whether the generated second generated image is real by distinguishing the monitoring original image from the second generated image. Whether the second generated image is real can be reflected by the loss function of the style original transfer model.
[0068] In the step S30, the first generated image is input into the style primitive transfer model, and the third generated image is output. Figure 2, the original monitoring image, the style image, the original style transfer model, the first generated image, the original style transfer model and the third generated image form a cyclic process, namely the first cyclic link. During the model training of the cyclic process, the original monitoring image is mapped to the style image and then mapped back to the third generated image. The third generated image has a large similarity with the original monitoring image, forming the first cyclic link. If X represents the original monitoring image, Y represents the style image, and the third generated image has a large similarity with the original monitoring image, expressed as X', then an XY-X' cyclic link is formed. Furthermore, the loss function of the first cyclic link can reflect the similarity between the third generated image X' and the original monitoring image X, thereby reflecting the feasibility of the monitoring privacy protection model described in the present invention.
[0069] Input the second generated image into the original style transfer model, and output the fourth generated image. Figure 2 , the style image, the original monitoring image, the original style transfer model, the second generated image, the original style transfer model and the fourth generated image form a cyclic process, namely the second cyclic link. During the model training of the cyclic process, the style image is simultaneously learned to be mapped to the original monitoring image and then mapped back to the fourth generated image. The fourth generated image has a large similarity with the style image, forming the second cyclic link. Furthermore, the loss function of the second cyclic link can reflect the similarity between the fourth generated image and the style image, thereby reflecting the feasibility of the monitoring privacy protection model of the present invention.
[0070] In S40, by adjusting the parameters of the original style transfer model and the original style transfer model involved in S20 to S30, the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial monitoring privacy protection model can be adjusted to iteratively optimize the monitoring privacy protection model and form a more accurate and stable monitoring privacy protection model. Thereby, the monitoring privacy protection model generates a more realistic privacy protection image and better protects the privacy of the monitoring video image.
[0071] In S50, the original monitoring image may be an original monitoring image of a specific application scenario. The original monitoring image is input into the stable monitoring privacy protection model, and the privacy protection image is output. The original monitoring image of a specific application scenario may refer to an original monitoring image in a certain application scenario. For example: an original monitoring image in a parking lot scenario, or an original monitoring image in a crossroad scenario, or an original monitoring image in a highway scenario, etc.
[0072] Through the privacy protection method of surveillance video images of the present invention, the stylized image set and the surveillance original image set are constructed, and the model optimization and iteration are performed according to the original style transfer model, the style original transfer model, the first loop link, the second loop link and the corresponding loss function to obtain the surveillance privacy protection model. Thus, the surveillance original image that needs to be style-transferred is sent to the surveillance privacy protection model to obtain the image after style transfer, that is, the privacy-protected image. The privacy-protected image is similar to the style image in terms of texture, color and other stylization aspects, and sensitive information such as character features and license plate numbers is desensitized, but the content and scene information of the surveillance original image are still retained, thereby improving the authenticity of the surveillance video image when it is used.
[0073] See also Figure 3 In one embodiment, the privacy protection method of the surveillance video image further includes:
[0074] S60: Replace the target image area of the privacy protection image with the target image area of the original monitoring image to obtain a privacy-protected monitoring video image.
[0075] In S60, the target image area of the privacy protection image refers to the feature area of the original surveillance image that needs to be retained. For example, the original surveillance image includes a target image area A that needs to be retained and an image area B that needs privacy protection. The target image area of the privacy protection image is replaced with the target image area A that needs to be retained, and the image area B that needs privacy protection is still the stylized image feature. Information that needs privacy protection may include sensitive information such as facial features of pedestrians or license plates of vehicles. Therefore, the original surveillance image not only retains the real target image area through the privacy protection method of the surveillance video image, but also protects the privacy-sensitive image area, thereby playing a role in protecting the privacy of other surrounding scenes.
[0076] In one embodiment, the style image set may be style images of the same creator, such as a set of cartoon-style images shot by the same director, or a set of illustration-style images shot by the same author, etc. Using a style image set of the same creator can make the style image set have the same style, which is conducive to stylizing the original surveillance image.
[0077] In one embodiment, the size of the style images in the style image set is the same as that of the surveillance original images in the surveillance original image set, which is conducive to combining the style image with the surveillance video image to form the privacy protection image.
[0078] In one embodiment, the S20 includes:
[0079] S210, inputting the monitored original image and the style image into an original style generator network, and outputting the first generated image;
[0080] S220, inputting the style image and the first generated image into an original style discriminator network for discriminating the style image and the first generated image;
[0081] The original style generator network and the original style discriminator network form the original style transfer model.
[0082] In S20, the original style transfer model can be constructed by Generative Adversarial Networks (GANs). The Generative Adversarial Network includes a generator network and a discriminator network, and the two networks are iteratively trained. Furthermore, based on the Generative Adversarial Network, the original style transfer model can realize applications such as image colorization, image style transfer, and image super-resolution.
[0083] In S210, the original style generator network may be a generator network for generating a stylized image, mapping the original monitoring image to the style image, and outputting the first generated image. The first generated image is a generated stylized image.
[0084] In S220, the original style discriminator network may be a discriminator network, which is used to discriminate between the style image and the first generated image, and then determine whether the generated first generated image (that is, the generated stylized image) is authentic. The original style generator network and the original style discriminator network adjust their parameters through mutual game, so that the loss function of the original style transfer model reaches the target balance point, thereby generating a better privacy protection image and achieving better protection of privacy information.
[0085] In one embodiment, in S210, the image feature size of the original monitoring image is 256*256*3 (H*W*C, Height*Width*Channel, height*width*number of channels). The original monitoring image passes through multiple convolutional layers, normalization layers, and nonlinear activation layers to form an image with a feature map size of 64*64*256. The image with a feature map size of 64*64*256 is further feature extracted, and the image feature size is restored to its original size, i.e., 256*256*3, through a residual convolution module and multiple deconvolution layers, thereby generating the first generated image (i.e., the generated stylized image).
[0086] In one embodiment, the residual convolution module includes a connection of multiple residual convolutional neural networks. The normalization layer includes an instance normalization layer or an adaptive instance normalization layer, etc. The nonlinear activation layer includes a nonlinear activation function such as a rectified linear unit (Rectified Linear Unit, ReLU).
[0087] In one embodiment, in S220, the original style discriminator network is used to discriminate the authenticity of the first generated image. The original style discriminator network is a binary classification network, and the classification target is to distinguish the data of the monitored original image from the data of the first generated image. The number of layers of the original style discriminator network is relatively shallow, and the image feature size input to the original style discriminator network is 256*256*3 (H*W*C, Height*Width*Channel, height*width*number of channels), and the feature map is downsampled after multiple convolution operations, and then the features are further classified using a binary classification loss function. The binary classification loss function includes a cross entropy loss function or a logarithmic loss function, etc.
[0088] In one embodiment, the S20 further includes:
[0089] S230, inputting the style image and the monitoring original image into a style original generator network, and outputting the second generated image;
[0090] S240, inputting the monitoring original image and the second generated image into a style original discriminator network for discriminating the monitoring original image and the second generated image;
[0091] The style primitive generator network and the style primitive discriminator network form the style primitive migration model.
[0092] In S20, the style primitive transfer model can be constructed by a generative adversarial network. Based on the generative adversarial network, the style primitive transfer model can realize applications such as image colorization, image style transfer, and image super-resolution.
[0093] In S230, the style original generator network may be a generator network for generating a stylized image, mapping the style image to the monitoring original image, and outputting the second generated image. The second generated image is a generated stylized image.
[0094] In S240, the style original discriminator network may be a discriminator network, which is used to discriminate between the monitoring original image and the second generated image, and then determine whether the generated second generated image (that is, the generated stylized image) is real. The style original generator network and the style original discriminator network adjust their parameters through mutual game, so that the loss function of the style original transfer model reaches the target balance point, thereby generating a better privacy protection image and achieving better protection of privacy information.
[0095] In one embodiment, the functional layers of the original style generator network in S230 are the same as the functional layers of the original style generator network in S210, and the input image feature size is also the same. For a specific description, reference may be made to the description of S210 in the above embodiment.
[0096] In one embodiment, the functional layers of the original style discriminator network in S240 are the same as the functional layers of the original style discriminator network in S220, and the input image feature size is also the same. For a specific description, reference may be made to the description of S220 in the above embodiment.
[0097] In one embodiment, in S30, the third generated image is obtained by inputting the first generated image into the style original generator network.
[0098] The first generated image is input into the style original generator network, and the third generated image is output. The monitoring original image, the style image, the original style generator network, the first generated image, the style original generator network and the third generated image form the first loop link. If X represents the monitoring original image, Y represents the style image, and the third generated image has a large similarity with the monitoring original image, represented by X', then an XY-X' loop link is formed.
[0099] In one embodiment, in S30, the fourth generated image is obtained by inputting the second generated image into the original style generator network.
[0100] The style image, the original monitoring image, the original style generator network, the second generated image, the original style generator network and the fourth generated image form the second loop link. The fourth generated image has a large similarity with the style image, expressed as Y', thus forming a YX-Y' loop link.
[0101] Therefore, the first loop link and the second loop link adopt the training method of Cycle Generative Adversarial Networks (CycleGAN). The first loop link uses unpaired training data and two mirror-symmetric generative adversarial networks to simultaneously learn the process of mapping the monitoring original image X to the style image Y and then mapping back to the third generated image X' and the process of mapping the style image Y to the monitoring original image X and then mapping back to the fourth generated image Y' during model training.
[0102] In one embodiment, the S40 includes:
[0103] S410, adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model according to the loss function of the surveillance privacy protection model;
[0104] Among them, the loss function of the monitoring privacy protection model is:
[0105]
[0106] Where X represents the original monitoring image, Y represents the style image, G represents the original style generator network, and D Y represents the original style discriminator network, F represents the original style generator network, D X represents the original style discriminator network, L GAN (G,D Y ,X,Y) represents the loss function of the original style transfer model, L GAN (F,D X ,Y,X) represents the loss function of the original style transfer model, λ is the weight parameter, represents the loss function of the first loop link, represents a loss function of the second loop link;
[0107] S420, adjusting the loss function of the monitoring privacy protection model to a target equilibrium point to form a stable monitoring privacy protection model;
[0108] The target balance point is:
[0109]
[0110] Among them, G * represents the weight of the original style discriminator network, F * represents the weight of the original generator network of the style, and the min-max function represents finding a discriminator network so that the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized, and the difference between the feature distribution of the input image and the generated distribution of the generator network is minimized.
[0111] In S410, the monitoring privacy protection model is trained and optimized by adjusting the loss function of the monitoring privacy protection model. GAN (G,D Y ,X,Y) can also be understood as the loss function of the original style generator network and the original style discriminator network. GAN (F,D X ,Y,X) can also be understood as the loss function of the style original generator network and the style original discriminator network. It can also be understood as a loss function of mapping the original monitoring image to the style image and then mapping back to the third generated image (which has a greater similarity to the original monitoring image). It can also be understood as a loss function of mapping the style image to the monitoring original image and then mapping it back to the fourth generated image (which has a greater similarity to the style image).
[0112] By adjusting the parameters of the original style generator network G and the original style discriminator network D Y The parameters of the original style generator network F and the original style discriminator network D X The parameters of L GAN (G,D Y ,X,Y)、L GAN (F,D X ,Y,X)、 , thereby adjusting the loss function of the monitoring privacy protection model.
[0113] In S420, the target balance point is the adjustment target of the loss function of the monitoring privacy protection model. The discriminator network in the min-max function includes the original style discriminator network and the style original discriminator network. The generator network in the min-max function includes the original style generator network and the style original generator network.
[0114] Specifically, the min-max function represents: finding the original style discriminator network so that the difference between the feature distribution of the monitored original image and the generated distribution of the original style generator network is maximized, and the difference between the feature distribution of the monitored original image and the generated distribution of the original style generator network is minimized; and finding the style original discriminator network so that the difference between the feature distribution of the style image and the generated distribution of the style original generator network is maximized, and the difference between the feature distribution of the style image and the generated distribution of the style original generator network is minimized.
[0115] By adjusting the parameters of the original style generator network G and the original style discriminator network D Y The parameters of the original style generator network F and the original style discriminator network D X The parameters of the monitoring privacy protection model are set so that the loss function of the monitoring privacy protection model reaches the target balance point, the monitoring original image is stylized, and sensitive information such as character features and license plates are desensitized and protected, while the target image area is replaced with the original image features, thereby retaining the authenticity of the monitoring original image.
[0116] In one embodiment, when looking for a target equilibrium point where the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized and the difference between the input image and the generated distribution of the generator network is minimized, functions such as KL divergence (Kullback-Leibler Divergence) and JS divergence (Jensen–Shannon Divergence) can also be used to describe the difference.
[0117] In one embodiment, the L loss function may be an L1 norm loss function or an L2 norm loss function, etc.
[0118] See also Figure 4 In one embodiment, the present invention provides a privacy protection device 100 for surveillance video images. The privacy protection device 100 for surveillance video images includes an acquisition module 10, an image generation module 20, a loop link module 30, a model optimization module 40, and a privacy protection image generation module 50. The acquisition module 10 is used to acquire a style image set and a surveillance original image set. The image generation module 20 is used to generate a first generated image and a second generated image according to the surveillance original image in the surveillance original image set and the style image in the style image set, wherein the first generated image is obtained by inputting the surveillance original image and the style image into an original style transfer model, and the second generated image is obtained by inputting the style image and the surveillance original image into the style original transfer model and outputting the style image and the surveillance original image.
[0119] The loop link module 30 is used to form a first loop link according to the original monitoring image, the first generated image and the third generated image, and to form a second loop link according to the style image, the second generated image and the fourth generated image, wherein the third generated image is obtained by inputting the first generated image into the style original migration model, and the fourth generated image is obtained by inputting the second generated image into the original style migration model. The model optimization module 40 is used to adjust the loss function of the original style migration model, the loss function of the style original migration model, the loss function of the first loop link and the loss function of the second loop link in the initial monitoring privacy protection model to form a stable monitoring privacy protection model. The privacy protection image generation module 50 is used to convert the original monitoring image into a privacy protection image according to the stable monitoring privacy protection model.
[0120] The acquisition module 10, the acquisition module 10, the image generation module 20, the loop link module 30, the model optimization module 40 and the privacy protection image generation module 50 in the privacy protection device 100 for monitoring video images correspond one-to-one to S10 to S50 in the privacy protection method for monitoring video images, respectively. For related descriptions, please refer to the description in the above embodiments.
[0121] In one embodiment, the image generation module 20 includes an original style transfer module (not shown in the figure). The original style transfer module is used to input the monitored original image and the style image into the original style generator network, output the first generated image, and input the style image and the first generated image into the original style discriminator network to discriminate the style image and the first generated image.
[0122] The original style transfer module corresponds to S210 and S220 one by one, and related descriptions may refer to the description in the above embodiment.
[0123] In one embodiment, the image generation module 20 further includes a style primitive migration module (not shown in the figure). The style primitive migration module is used to input the style image and the monitoring primitive image into the style primitive generator network, output the second generated image, and input the monitoring primitive image and the second generated image into the style primitive discriminator network to discriminate the monitoring primitive image and the second generated image.
[0124] The style original transfer module corresponds to S230 and S240 one by one, and the related description can refer to the description in the above embodiment.
[0125] In one embodiment, the third generated image in the loop link module is obtained by inputting the first generated image into the style original generator network. For related description, reference may be made to the related description in S30 in the above embodiment.
[0126] In one embodiment, in the loop link module, the fourth generated image is obtained by inputting the second generated image into the original style generator network. For related description, reference may be made to the related description in S30 in the above embodiment.
[0127] In one embodiment, the model optimization module 40 includes a parameter adjustment module (not shown in the figure) and a model generation module (not shown in the figure). The parameter adjustment module is used to adjust the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial monitoring privacy protection model according to the loss function of the monitoring privacy protection model. The model generation module is used to adjust the loss function of the monitoring privacy protection model to the target balance point to form a stable monitoring privacy protection model.
[0128] Among them, the loss function of the monitoring privacy protection model is:
[0129]
[0130] Where X represents the original monitoring image, Y represents the style image, G represents the original style generator network, and D Y represents the original style discriminator network, F represents the original style generator network, D X represents the original style discriminator network, L GAN (G,D Y ,X,Y) represents the loss function of the original style transfer model, L GAN (F,D X ,Y,X) represents the loss function of the original style transfer model, λ is the weight parameter, represents the loss function of the first loop link, Denotes the loss function of the second loop link.
[0131] The target balance point is:
[0132]
[0133] Among them, G * represents the weight of the original style discriminator network, F *represents the weight of the original generator network of the style, and the min-max function represents finding a discriminator network so that the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized, and the difference between the feature distribution of the input image and the generated distribution of the generator network is minimized.
[0134] The privacy protection image generation module is used to input the original monitoring image into the stable monitoring privacy protection model and output the privacy protection image.
[0135] The parameter adjustment module corresponds to the S410 one by one, and the relevant description can refer to the description in the above embodiment. The model generation module corresponds to the S420 one by one, and the relevant description can refer to the description in the above embodiment.
[0136] See also Figure 4 In one embodiment, the surveillance video image privacy protection device 100 further includes a surveillance video image generation module 60. The surveillance video image generation module 60 is used to replace the target image area of the privacy protection image with the target image area of the original surveillance image to obtain a surveillance video image after privacy protection.
[0137] The monitoring video image generation module 60 corresponds to the S60 one by one, and the related description can refer to the description in the above embodiment.
[0138] In one embodiment, the acquisition module 10, the image generation module 20, the loop link module 30, the model optimization module 40, the privacy protection image generation module 50 and the monitoring video image generation module 60 may include but are not limited to a central processing unit (CPU), an embedded microcontroller (MCU), an embedded microprocessor (MPU), and an embedded system on chip (SoC).
[0139] In one embodiment, the present invention provides a computer device, including a processor and a memory for storing a computer program. The processor is used to execute the program code stored in the memory to implement the method steps described in any of the above embodiments. The computer device may be different according to different configurations or performances. The computer device also includes a memory. At least one instruction is stored in the memory, and at least one instruction is loaded and executed by the processor to implement the data provided by each method in each of the above embodiments. The computer device may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The computer device may also include other components for implementing device functions, which will not be described in detail here.
[0140] In one embodiment, the present invention provides a computer-readable storage medium. The computer-readable storage medium stores program code, and when the computer program is executed by a processor, the method steps of any of the above embodiments are implemented. The computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0141] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0142] In one embodiment, the privacy protection method and the privacy protection device for monitoring video images can be applied to scenes such as viewing monitoring video images and providing vehicle owners with evidence of vehicle violations. The privacy protection method and the privacy protection device for monitoring video images can be used to protect the privacy area outside the target image area of the original monitoring image and present a real target image.
[0143] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A privacy protection method for monitoring video images, characterized in that: include: Obtain a style image set and monitor the original image set; Generate a first generated image and a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set, wherein the first generated image is obtained by inputting the monitored original image and the style image into the original style transfer model, and the second generated image is obtained by inputting the style image and the monitored original image into the style original transfer model; A first loop link is formed according to the monitored original image, the style image, the original style transfer model, the first generated image, the style original transfer model, and the third generated image, and a second loop link is formed according to the style image, the monitored original image, the style original transfer model, the second generated image, the original style transfer model, and the fourth generated image, wherein the third generated image is obtained by inputting the first generated image into the style original transfer model, and the fourth generated image is obtained by inputting the second generated image into the original style transfer model; Adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model to form a stable surveillance privacy protection model; The step of adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial monitoring privacy protection model to form a stable monitoring privacy protection model includes: According to the loss function of the monitoring privacy protection model, adjusting the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial monitoring privacy protection model; Among them, the loss function of the monitoring privacy protection model is: X represents the original monitoring image, Y represents the style image, G represents the original style generator network, and D Y represents the original style discriminator network, F represents the original style generator network, and D X Represents the original style discriminator network, L GAN (F,D Y , Y, X) represents the loss function of the original style transfer model, L GAN (F,D X , Y, X) represents the loss function of the original style transfer model, λ is the weight parameter, represents the loss function of the first loop link, represents a loss function of the second loop link; Adjusting the loss function of the monitoring privacy protection model to a target equilibrium point to form a stable monitoring privacy protection model; The target balance point is: Wherein, G* represents the weight of the original style discriminator network, F* represents the weight of the original style generator network, and the min-max function represents finding a discriminator network so that the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized, and the difference between the feature distribution of the input image and the generated distribution of the generator network is minimized; The original surveillance image is converted into a privacy-preserving image according to the stable surveillance privacy-preserving model.
2. The privacy protection method for surveillance video images according to claim 1, characterized in that: Generating a first generated image according to the monitored original image in the monitored original image set and the style image in the style image set, including: Inputting the monitored original image and the style image into an original style generator network, and outputting the first generated image; Inputting the style image and the first generated image into an original style discriminator network to discriminate the style image and the first generated image; The original style generator network and the original style discriminator network form the original style transfer model.
3. The privacy protection method for surveillance video images according to claim 2, characterized in that: Generating a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set, comprising: Inputting the style image and the monitoring original image into a style original generator network, and outputting the second generated image; Inputting the monitoring original image and the second generated image into a style original discriminator network to discriminate the monitoring original image and the second generated image; The style primitive generator network and the style primitive discriminator network form the style primitive migration model.
4. The privacy protection method for surveillance video images according to claim 3 is characterized in that: A first loop link is formed according to the monitoring original image, the first generated image and the third generated image, wherein the third generated image is obtained by inputting the first generated image into the style original generator network.
5. The privacy protection method for surveillance video images according to claim 4, characterized in that: A second loop link is formed according to the style image, the second generated image and the fourth generated image, wherein the fourth generated image is obtained by inputting the second generated image into the original style generator network.
6. The privacy protection method for surveillance video images according to claim 1, characterized in that: Also includes: The target image area of the privacy protection image is replaced with the target image area of the original monitoring image to obtain a privacy-protected monitoring video image.
7. A privacy protection device for monitoring video images, characterized in that: include: An acquisition module is used to acquire a style image set and monitor an original image set; An image generation module, configured to generate a first generated image and a second generated image according to the monitored original image in the monitored original image set and the style image in the style image set, wherein the first generated image is obtained by inputting the monitored original image and the style image into the original style transfer model, and the second generated image is obtained by inputting the style image and the monitored original image into the style original transfer model; a loop link module, configured to form a first loop link according to the monitoring original image, the first generated image, and a third generated image, and to form a second loop link according to the style image, the second generated image, and a fourth generated image, wherein the third generated image is obtained by inputting the first generated image into the style original migration model, and the fourth generated image is obtained by inputting the second generated image into the original style migration model; A model optimization module, used to adjust the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model to form a stable surveillance privacy protection model; A privacy protection image generation module, used for converting the original monitoring image into a privacy protection image according to the stable monitoring privacy protection model; The model optimization module includes: A parameter adjustment module, configured to adjust the loss function of the original style transfer model, the loss function of the original style transfer model, the loss function of the first loop link, and the loss function of the second loop link in the initial surveillance privacy protection model according to the loss function of the surveillance privacy protection model; Among them, the loss function of the monitoring privacy protection model is: X represents the original monitoring image, Y represents the style image, G represents the original style generator network, and D Y represents the original style discriminator network, F represents the original style generator network, and D X Represents the original style discriminator network, L GAN (G,D Y , X, Y) represents the loss function of the original style transfer model, L GAN (F,D X , Y, X) represents the loss function of the original style transfer model, λ is the weight parameter, represents the loss function of the first loop link, represents a loss function of the second loop link; Adjusting the loss function of the monitoring privacy protection model to a target equilibrium point to form a stable monitoring privacy protection model; The target balance point is: Among them, G* represents the weight of the original style discriminator network, F* represents the weight of the style original generator network, and the min-max function represents finding a discriminator network so that the difference between the feature distribution of the input image and the generated distribution of the generator network is maximized, and the difference between the feature distribution of the input image and the generated distribution of the generator network is minimized.
8. The privacy protection device for surveillance video images according to claim 7, characterized in that: The image generation module comprises: The original style transfer module is used to input the monitored original image and the style image into the original style generator network, output the first generated image, and input the style image and the first generated image into the original style discriminator network to discriminate the style image and the first generated image.
9. The privacy protection device for surveillance video images according to claim 8, characterized in that: The image generation module also includes: The style primitive transfer module is used to input the style image and the monitoring primitive image into the style primitive generator network, output the second generated image, and input the monitoring primitive image and the second generated image into the style primitive discriminator network to discriminate the monitoring primitive image and the second generated image.
10. The privacy protection device for surveillance video images according to claim 9, characterized in that: The loop link The third generated image in the module is obtained by inputting the first generated image into the style original generator network.
11. The privacy protection device for surveillance video images according to claim 10, characterized in that: In the loop link module, the fourth generated image is obtained by inputting the second generated image into the original style generator network.
12. The privacy protection device for surveillance video images according to claim 7, characterized in that: Also includes: The monitoring video image generation module is used to replace the target image area of the privacy protection image with the target image area of the monitoring original image to obtain the monitoring video image after privacy protection.
