Image protection method, device and storage medium

Abstract

The application provides an image protection method and device and a storage medium, relates to the technical field of image processing, and can solve the problem that image information is destroyed and thus the image cannot be recognized. The method comprises the following steps: obtaining a first face image from an original image; performing image analysis on the first face image through a face network model to obtain edge information of the first face image, the edge information being used to represent the contour of each face part in the first face image; performing image coding on the first face image and the edge information of the first face image through a variational autoencoder to obtain a second face image; and updating the first face image in the original image to the second face image to obtain a target image. The embodiment of the application is used in the process of face image privacy protection in face recognition.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image protection method, apparatus and storage medium. Background Technology

[0002] With the continuous improvement of 5G technology, facial recognition technology has permeated every corner of people's lives, and the protection of privacy during the facial recognition process is an urgent problem to be solved.

[0003] The current facial recognition process involves uploading images captured by electronic devices to the cloud for recognition. To protect privacy, a mosaic is added to the captured images to obscure parts of the image, thereby achieving the purpose of protecting privacy.

[0004] However, while the above solution can protect user privacy, adding a mosaic will destroy the image information, making the image unrecognizable. Summary of the Invention

[0005] This application provides an image protection method, apparatus, and storage medium that can solve the problem of images becoming unrecognizable due to the destruction of image information.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] In a first aspect, this application provides an image protection method, which includes: obtaining a first facial image from an original image; performing image parsing on the first facial image using a facial network model to obtain edge information of the first facial image, the edge information being used to characterize the contours of various facial parts in the first facial image; performing image encoding on the first facial image and the edge information of the first facial image using a variational autoencoder to obtain a second facial image; and updating the first facial image in the original image with the second facial image to obtain a target image.

[0008] Based on the above technical solution, the image protection method provided in this application embodiment can first obtain a first facial image from the original image; then, through a facial network model, perform image parsing on the first facial image to obtain edge information of the first facial image, which is used to characterize the contours of each facial part in the first facial image; next, through a variational autoencoder, perform image encoding on the first facial image and the edge information of the first facial image to obtain a second facial image; finally, update the first facial image in the original image to the second facial image to obtain the target image. Since the above image parsing can obtain the contours of each facial part in the facial image, when the variational autoencoder is used for encoding, only the content of each facial part in the facial image is encoded, while the contours of each facial part remain unchanged. Therefore, it is ensured that the generated second facial image, while scrambling the image content of the first facial image, also has the attribute features of the first facial image, so that the final target image protects user privacy and is identifiable.

[0009] In a first possible implementation of the first aspect, obtaining the first facial image from the original image includes: performing image detection on the original image to obtain coordinate information of each pixel in the original image; and segmenting the original image based on the coordinate information of each pixel to obtain the first facial image.

[0010] In the second possible implementation of the first aspect, the above-mentioned image parsing of the first facial image through a facial network model to obtain the edge information of the first facial image includes: performing image segmentation on the first facial image through the facial network model to obtain N region blocks, each region block corresponding to a facial part in the first facial image; obtaining the edge information of each of the N region blocks, and determining the edge information of the N region blocks as the edge information of the first facial image.

[0011] In a third possible implementation of the first aspect, the variational autoencoder includes an encoding module and a generation module; the method of image encoding the first facial image and its edge information using the variational autoencoder to obtain a second facial image includes: encoding the first facial image using the encoding module to obtain semantic feature information, which is used to characterize the image content and image features of each facial part in the first facial image; fusing the semantic feature information with the edge information of the first facial image to obtain identity encoding information and attribute encoding information of the first facial image, whereby the identity encoding information is used to characterize the encoding result of the image content of each facial part in the first facial image, and the attribute encoding information is used to characterize the encoding result of the image features of each facial part in the first facial image; encrypting the identity encoding information of the first facial image to obtain target identity encoding information; and generating the second facial image using the generation module based on the attribute encoding information and the target identity encoding information of the first facial image.

[0012] In a fourth possible implementation of the first aspect, updating the first facial image in the original image to the second facial image to obtain the target image includes: obtaining the coordinate information of each pixel of the first facial image; replacing the pixels of the first facial image with the pixels of the second facial image according to the coordinate information of each pixel of the first facial image; and smoothing the pixels of the second facial image and the pixels of the original image to obtain the target image.

[0013] Secondly, this application provides an image protection device, which includes: an acquisition unit, a parsing unit, an encoding unit, and a processing unit, wherein: the acquisition unit is used to acquire a first facial image from an original image; the parsing unit is used to perform image parsing on the first facial image acquired by the acquisition unit using a facial network model to obtain edge information of the first facial image, the edge information being used to characterize the contours of each facial part in the first facial image; the encoding unit is used to perform image encoding on the first facial image acquired by the acquisition unit and the edge information of the first facial image parsed by the parsing unit using a variational autoencoder to obtain a second facial image; and the processing unit is used to update the first facial image in the original image with the second facial image to obtain a target image.

[0014] In a first possible implementation of the second aspect, the aforementioned acquisition unit is specifically used to: perform image detection on the aforementioned original image to obtain the coordinate information of each pixel in the aforementioned original image; and segment the aforementioned original image based on the coordinate information of each pixel to obtain the aforementioned first facial image.

[0015] In a second possible implementation of the second aspect, the parsing unit is specifically used to: perform image segmentation on the first facial image using the facial network model to obtain N region blocks, each region block corresponding to a facial part in the first facial image; obtain the edge information of each of the N region blocks, and determine the edge information of the N region blocks as the edge information of the first facial image.

[0016] In a third possible implementation of the second aspect, the variational autoencoder includes an encoding module and a generation module. Specifically, the encoding unit is used to: encode the first facial image using the encoding module to obtain semantic feature information, which characterizes the image content and features of each facial part in the first facial image; fuse the semantic feature information with the edge information of the first facial image to obtain identity encoding information and attribute encoding information of the first facial image, whereby the identity encoding information characterizes the encoding result of the image content of each facial part in the first facial image, and the attribute encoding information characterizes the encoding result of the image features of each facial part in the first facial image; encrypt the identity encoding information of the first facial image to obtain target identity encoding information; and generate the second facial image using the generation module based on the attribute encoding information and the target identity encoding information of the first facial image.

[0017] In the fourth possible implementation of the second aspect, the processing unit is specifically used to: obtain the coordinate information of each pixel of the first facial image; replace the pixels of the first facial image with the pixels of the second facial image according to the coordinate information of each pixel of the first facial image; and smooth the pixels of the second facial image and the pixels of the original image to obtain the target image.

[0018] Thirdly, this application provides an image protection device, which includes: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is used to run computer programs or instructions to implement the image protection method as described in the first aspect and any possible implementation of the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a terminal, cause the terminal to perform the image protection method as described in the first aspect and any possible implementation thereof.

[0020] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on an image protection device, cause the image protection device to perform the image protection method as described in the first aspect and any possible implementation thereof.

[0021] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run computer programs or instructions to implement the image protection method as described in the first aspect and any possible implementation thereof.

[0022] Specifically, the chip provided in this application embodiment also includes a memory for storing computer programs or instructions. Attached Figure Description

[0023] Figure 1 A flowchart illustrating an image protection method provided in this application embodiment;

[0024] Figure 2 This is a network structure diagram of the encoding module provided in an embodiment of this application;

[0025] Figure 3 This is one of the internal flowcharts of an image protection method provided in an embodiment of this application;

[0026] Figure 4 A schematic diagram illustrating the training process of the variational autoencoder provided in an embodiment of this application;

[0027] Figure 5 This is a second internal flowchart of an image protection method provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an image protection device provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of another image protection device provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0031] The image protection method, apparatus, and storage medium provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0032] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0033] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0034] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0035] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0036] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0037] With the continuous improvement of 5G technology, more and more intelligent systems are beginning to utilize cloud computing services, leveraging the powerful computing capabilities of cloud servers to handle large-scale, high-consumption tasks, thereby alleviating the computing pressure on terminals.

[0038] For example, facial recognition technology has permeated every corner of people's lives. This means that facial information is not just a simple identifier between people, but also records more private information such as an individual's whereabouts, giving it significant privacy value. Currently, large amounts of unprotected facial information are freely disseminated on the public internet, posing a huge security risk. Privacy breaches can lead to intimidation, harassment, fraud, and even irreparable damage to individuals and society.

[0039] Currently, there are two common methods and systems for protecting facial privacy:

[0040] 1) A privacy and security solution for facial recognition based on deep learning and encryption algorithms.

[0041] For example, the above-mentioned face recognition privacy and security scheme based on deep learning and encryption algorithms may include the following steps A and B:

[0042] A. Use deep learning algorithms to extract facial features.

[0043] B. Encryption algorithms are used for facial recognition encryption.

[0044] 2) Enhance the communication process and protect facial information during transmission to minimize the transmission of unauthenticated or potentially security-risk facial information.

[0045] In existing technologies, facial recognition systems primarily acquire images through terminal devices and upload them to the cloud for recognition, based on the identification of real, full-color faces within the images. However, most existing privacy protection methods destroy the original image information, rendering the images unrecognizable and unable to meet recognition requirements. For example, while adding mosaics to images can protect user privacy, it destroys the image's structural information, resulting in a loss of recognizability and rendering it unusable in facial recognition systems. Alternatively, it may introduce numerous complex encryption algorithms and communication processes, increasing additional computation and costs. Especially when dealing with large-scale facial recognition data, existing privacy protection methods cannot be directly applied to real-world scenarios. Integrating encryption algorithms with existing facial recognition systems requires a significant reconstruction of the entire recognition process, leading to substantial changes and complex deployment. Therefore, improvements to existing facial recognition algorithms are necessary.

[0046] To address the problem of image unrecognization caused by image information corruption in existing technologies, this application provides an image protection method. First, a first facial image is obtained from the original image. Then, a facial network model is used to analyze the first facial image to obtain its edge information, which represents the contours of various facial features. Next, a variational autoencoder is used to encode the first facial image and its edge information to obtain a second facial image. Finally, the first facial image in the original image is updated with the second facial image to obtain the target image. Because the image analysis obtains the contours of various facial features, the subsequent encoding using the variational autoencoder only encodes the content of each facial feature, while the contours remain unchanged. Therefore, the generated second facial image, while scrambling the image content of the first facial image, retains the attribute features of the first facial image, thus ensuring that the final target image protects user privacy while maintaining recognizability.

[0047] like Figure 1The diagram shows a flowchart of an image protection method provided in an embodiment of this application. The method includes the following steps S101 to S104:

[0048] S101. Obtain the first facial image from the original image.

[0049] In this embodiment of the application, the original image may be an image stored inside the electronic device, an image captured in real time by the electronic device through a camera, or a video frame of a video stored inside the electronic device.

[0050] For example, the original image described above includes at least one face image and a background image.

[0051] In this embodiment of the application, the first facial image can be a human face image.

[0052] For example, the first facial image mentioned above is a face image that needs to be protected.

[0053] S102. Using a facial network model, perform image analysis on the first facial image to obtain the edge information of the first facial image.

[0054] In this embodiment of the application, the above-described facial network model is used to segment facial images into multiple different types.

[0055] For example, the above-mentioned face network model can be based on a real-time semantic segmentation network (BiSeNet) and then trained on a large-scale face image dataset (CelebAMask-HQ).

[0056] In this embodiment of the application, the aforementioned edge information is used to characterize the contours of each facial part in the aforementioned first facial image.

[0057] For example, the aforementioned facial features may include: ears, nose, eyes, eyebrows, mouth, lips, etc.

[0058] S103. The second face image is obtained by image encoding the first face image and its edge information through a variational autoencoder.

[0059] In this embodiment of the application, the variational autoencoder is used to re-encode and decode facial images to generate entirely new facial images.

[0060] In this embodiment of the application, the above-described image encoding process is used to encrypt facial images.

[0061] In this embodiment of the application, the electronic device can re-encode the face image and the outline of the face in the face image to change the identity information of the image in the face image, thereby obtaining a completely new face image.

[0062] S104. Update the first face image in the original image to the second face image to obtain the target image.

[0063] In this embodiment of the application, after obtaining a new face image, the electronic device can replace the face image in the original image with the new face image to obtain an encrypted target image.

[0064] In the image protection method provided in this application embodiment, a first facial image can be obtained from the original image first; then, the first facial image is parsed using a facial network model to obtain edge information of the first facial image, which is used to characterize the contours of each facial part in the first facial image; next, the first facial image and the edge information of the first facial image are image encoded using a variational autoencoder to obtain a second facial image; finally, the first facial image in the original image is updated to the second facial image to obtain the target image. Since the above image parsing can obtain the contours of each facial part in the facial image, when the variational autoencoder is used for subsequent encoding, only the content of each facial part in the facial image is encoded, while the contours of each facial part remain unchanged. Therefore, it is ensured that the generated second facial image, while scrambling the image content of the first facial image, also has the attribute features of the first facial image, so that the final target image protects user privacy and is identifiable.

[0065] Optionally, in this embodiment of the application, step S101 may include the following steps S101a and S101b:

[0066] S101a. Perform image detection on the original image to obtain the coordinate information of each pixel in the original image.

[0067] In the embodiments of this application, the electronic device may use a face detection algorithm (MTCNN) or a face analysis framework (DeepFace, FaceLib) to detect the original image.

[0068] For example, electronic devices can use face detection algorithms or face analysis frameworks with minimal parameters for detection.

[0069] It should be noted that the smaller the above parameters are, the lower the requirements for the video memory of the electronic device.

[0070] In this embodiment of the application, the coordinate information is used to indicate the coordinate position of each object in the original image.

[0071] For example, an electronic device can obtain the bounding box coordinates of each object in the original image by detecting it.

[0072] S101b: Based on the coordinate information of each pixel, the original image is segmented to obtain the first facial image.

[0073] In this embodiment of the application, after obtaining the bounding box coordinates of each object in the original image, the electronic device can again use the face detection algorithm described above to filter out the coordinate information of the pixels of the first face image from the coordinate information of each pixel. Then, based on the coordinate information of the pixels of the first face image, the face image and the background image in the original image are cropped to obtain the first face image.

[0074] Thus, by using a face detection algorithm with a small number of parameters, the first face image is segmented from the original image based on the pixel coordinate information. Subsequent processing is then performed only on the first face image, which not only reduces the cost of the required electronic equipment but also improves the accuracy of the obtained first face image.

[0075] Optionally, in this embodiment of the application, step S102 may include the following steps S102a and S102b:

[0076] S102a. Using a facial network model, the first facial image is segmented to obtain N region blocks.

[0077] Where N is a positive integer.

[0078] In this embodiment of the application, each of the above N region blocks corresponds to a facial part in the above first facial image.

[0079] For example, the facial features corresponding to the above N region blocks may include skin, nose, eyes, eyebrows, ears, mouth, lips, and head.

[0080] In this embodiment of the application, each of the above N region blocks may also correspond to an image region in the above first facial image.

[0081] For example, the N region blocks may also include image regions containing parts such as hats, glasses, earrings, necklaces, necks, and clothes.

[0082] S102b: Obtain the edge information of each of the N region blocks, and determine the edge information of the N region blocks as the edge information of the first facial image.

[0083] For example, the electronic device can use a facial network model to classify the face image into 19 categories, including facial parts (such as skin, nose, eyes, eyebrows, ears, mouth, lips, head) and other parts (such as hat, glasses, earrings, necklace, neck and clothing, etc.); then the segmented results are processed to obtain a face mask, and an edge extraction algorithm is used to obtain the face edge image (i.e. the edge information of the first facial image mentioned above).

[0084] In this embodiment of the application, the electronic device can use a deep network to perform face segmentation and obtain face parsing results (i.e., the aforementioned edge information), which include face boundary and face contour information.

[0085] For example, the process of using a deep network for face segmentation described above is as follows: the first facial image is segmented into regions with semantic meaning, and the semantic category of each region is identified, realizing a semantic reasoning process from the bottom layer to the top layer, and finally obtaining a result with semantic annotation.

[0086] In this way, by dividing the face image into multiple regions according to different facial features, and then obtaining the contour of the corresponding part of each region, the efficiency and accuracy of obtaining the contour of the first facial image are improved.

[0087] Optionally, in this embodiment of the application, the variational autoencoder includes an encoding module and a generation module; step S103 may include the following steps S103a to S103d:

[0088] S103a. The first facial image is encoded by the encoding module to obtain semantic feature information.

[0089] In this embodiment of the application, the above-mentioned encoding module is used to encode the image.

[0090] For example, the internal structure of the above-mentioned encoding module can be a U-net structure.

[0091] In this embodiment of the application, the semantic feature information is used to characterize the image content and image features of each facial part in the first facial image.

[0092] For example, the semantic feature information mentioned above can be a semantic feature vector.

[0093] S103b: The semantic feature information is fused with the edge information of the first facial image to obtain the identity encoding information and attribute encoding information of the first facial image.

[0094] In this embodiment of the application, the aforementioned identity encoding information is used to characterize the encoding result of the image content of each facial part in the aforementioned first facial image.

[0095] For example, the aforementioned identity encoding information can be the identity information of an image.

[0096] In this embodiment of the application, the attribute encoding information is used to characterize the encoding results of the image features of each facial part in the first facial image.

[0097] For example, the attribute encoding information mentioned above may include image actions, image expressions, etc.

[0098] Example 1, such as Figure 2 The diagram shown is a network structure diagram of the aforementioned encoding module. Taking the first facial image as F... t The edge information of the first facial image is F e For example, the electronic device first processes the first facial image F... t The input is fed into the encoding module of the U-net structure, where it is reshaped into a two-dimensional vector. Then, the semantic feature vector is normalized using a regularization algorithm (L2 Norm). The normalized semantic feature vector is then subjected to a probability transformation using an activation function (Softmax) to obtain probability values. Finally, a channel-level attention mechanism is used to coordinate global information; this is then combined with edge information F... e The processed result is element-wise added, then passed through two linear layers with activation functions (ReLU), followed by a channel-dimension splitting (Splitgate) operation and further reshaping (Reshape) to obtain the variable μ. id and δ id And attribute encoding information α; finally, the variables μ and δ obtained by the encoding module of the variational autoencoder are used as mean vector and variance vector, respectively, and then reparameterized to obtain the latent variable z after cornering (i.e. the above identity encoding information).

[0099] It should be noted that the above variable μ id δ id Both the attribute encoding information α and the attribute encoding information α are one-dimensional vectors.

[0100] S103c: Encrypt the identity encoding information of the first facial image to obtain the target identity encoding information.

[0101] In this embodiment of the application, the aforementioned target identity information is the identity information of a newly generated face image after encryption.

[0102] Example 2, in conjunction with Example 1 above, after obtaining the aforementioned latent variable z, the electronic device can perform directional processing (such as fuzzing, perturbation, and mixing) on ​​the latent variable z to obtain a new latent variable z' (i.e., the aforementioned target identity encoding information).

[0103] S103d: The generation module generates a second facial image based on the attribute encoding information and target identity encoding information of the first facial image.

[0104] In this embodiment of the application, the above-mentioned generation module is used to decode the encoded information to generate an image.

[0105] Example 3, combined with Example 2 above, such as Figure 3 As shown, after obtaining the aforementioned latent variable z' and the aforementioned attribute encoding information α, the electronic device can input the connection vector of the latent variable z' and the attribute encoding information α into the generation module G for decoding, so as to output a synthetic face (i.e., the aforementioned second facial image).

[0106] In this way, by changing only the identity encoding information in the face image while keeping the original attribute encoding information unchanged, the generated second face image retains the expression, posture and other attributes of the first face image, thus ensuring the recognizability of the face image while achieving privacy protection.

[0107] The training process of the variational autoencoder involved in the embodiments of this application is illustrated below:

[0108] For example, such as Figure 4 As shown, the training data includes the target face F t and edge information F e And F corresponding to different images of the same identity t’ and F e’ For example, the above training process may include the following steps A1 to A4:

[0109] A1. Encoding module E will input the target face F t and edge information F e Encoded as variable μ id and δ id And attribute encoding α; F t’ and F e’ The corresponding value is obtained using Formula 1, which is as follows:

[0110] μ id ,δ id ,α=E(F t ,F e ) Formula 1

[0111] A2, variable μ id and δ id After reparameterization, the latent layer variable z is obtained through Equation 2, where ε ~ N(0,I). Equation 2 is as follows:

[0112] z=∈δ id +μid Formula 2

[0113] A3. The latent variable z undergoes a fine-tuning operation, generating z' using Formula 3, which changes the identity code. Formula 3 is as follows:

[0114] z'=M(z) Formula 3

[0115] A4. After concatenating the latent variables z and z' with the attribute code α, the results are fed into the generation module G. The synthesized face is obtained using formulas 4 and 5, which are as follows:

[0116] Y t =G(concat(z,α)) Formula 4

[0117] Y p =G(concat(z',α)) Formula 5

[0118] It should be noted that the target loss function L in the above training process can be obtained through Equation 6, as follows:

[0119] L = L mse +α1L reg +α2L perc +α3L id +α4L intra Formula 6

[0120] Among them, L mse To reconstruct the loss, the variational autoencoder is designed to generate high-fidelity face images while ensuring consistency in face images generated by the same identity encoding. Here, N is the number of samples, Mx is the number of pixels in the samples, and L... mse It can be obtained through Formula 7, which is as follows:

[0121]

[0122] Among them, L reg Let L be the KL divergence loss, and Mz be the vector dimension of the latent variable z. reg This can be obtained using Formula 8, which is as follows:

[0123]

[0124] Among them, L intra The intra-class loss is used to ensure that different face images of the same identity can obtain the same encoded vector after being processed by the encoding module. intra This can be obtained using Formula 9, which is as follows:

[0125]

[0126] Among them, L perc The perceptual loss is used to measure the semantic consistency of different synthetic faces, where φ l The L represents the activation values ​​of the ReLU4_2 layer in the pre-trained model VGG. perc This can be obtained using Formula 10, which is as follows:

[0127]

[0128] Among them, L id It is an identity consistency loss, used to reduce the consistency between the synthesized face after altered encoding and the original encoded face, thereby achieving a confidentiality effect. id (·) represents the deep features of the pre-trained face recognition system, <·> represents the cosine similarity, and τ is a set threshold to protect the authenticity of the image. id It can be obtained through Formula 11, as follows:

[0129]

[0130] Where, α i This is a coordination coefficient used to ensure generation performance.

[0131] In the embodiments of this application, the image protection method provided by this application decouples facial features by using a trained variational autoencoder and processes identity encoding separately to ensure the stability and specificity of the generated second facial image, thereby ensuring that the second facial image is identifiable.

[0132] Optionally, in this embodiment of the application, step S104 may include the following steps S104a to S104c:

[0133] S104a. Obtain the coordinate information of each pixel in the first facial image.

[0134] In this embodiment of the application, the electronic device can perform image detection on the first facial image to obtain the coordinate positions of each pixel that makes up the first facial image.

[0135] S104b: Based on the coordinate information of each pixel in the first facial image, replace the pixels of the first facial image with the pixels of the second facial image.

[0136] For example, such as Figure 5 As shown, the electronic device can replace all pixels of the target face (i.e., the first face image mentioned above) in the original image with pixels of the synthesized face (i.e., the second face image mentioned above), while the pixels of the background image other than the first face image remain unchanged, so as to achieve privacy protection of the original image.

[0137] S104c: Smooth the pixels of the second facial image and the pixels of the original image to obtain the target image.

[0138] In this embodiment of the application, after the electronic device replaces the pixels of the first facial image with the pixels of the second facial image, it can smooth the pixels in the contact area between the pixels of the second facial image and the original image to obtain the final target image.

[0139] Thus, by replacing the face image that needs protection with an encrypted face image and then smoothing the image after replacement, the image quality and recognizability of the obtained target image are improved.

[0140] Optionally, in the embodiments of this application, the above-mentioned image protection method processes the image after it is acquired, rather than introducing a large amount of computation during the recognition process like existing encryption algorithms. Therefore, it not only reduces the computational cost, but can also be combined with most existing face recognition systems.

[0141] This application embodiment can divide the image protection device into functional modules or functional units according to the above method examples. For example, each function can be divided into a separate functional module or functional unit, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or in software functional modules or functional units. The module or unit division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0142] like Figure 6 The diagram shown is a structural schematic of an image protection device provided in an embodiment of this application. The device includes: an acquisition unit 201, a parsing unit 202, an encoding unit 203, and a processing unit 204.

[0143] The acquisition unit 201 is used to acquire a first facial image from the original image; the parsing unit 202 is used to perform image parsing on the first facial image acquired by the acquisition unit through a facial network model to obtain edge information of the first facial image, which is used to characterize the contour of each facial part in the first facial image; the encoding unit 203 is used to perform image encoding on the edge information of the first facial image acquired by the acquisition unit and the first facial image parsed by the parsing unit through a variational autoencoder to obtain a second facial image; and the processing unit 204 is used to update the first facial image in the original image with the second facial image to obtain a target image.

[0144] Optionally, in this embodiment of the application, the acquisition unit 201 is specifically used to: perform image detection on the original image to obtain the coordinate information of each pixel in the original image; and segment the original image according to the coordinate information of each pixel to obtain the first facial image.

[0145] Optionally, in this embodiment of the application, the parsing unit 202 is specifically used to: perform image segmentation on the first facial image using the facial network model to obtain N region blocks, each region block corresponding to a facial part in the first facial image; obtain the edge information of each of the N region blocks, and determine the edge information of the N region blocks as the edge information of the first facial image.

[0146] Optionally, in this embodiment, the variational autoencoder includes an encoding module and a generation module; the encoding unit 203 is specifically used for: encoding the first facial image through the encoding module to obtain semantic feature information, which is used to characterize the image content and image features of each facial part in the first facial image; fusing the semantic feature information with the edge information of the first facial image to obtain identity encoding information and attribute encoding information of the first facial image, whereby the identity encoding information is used to characterize the encoding result of the image content of each facial part in the first facial image, and the attribute encoding information is used to characterize the encoding result of the image features of each facial part in the first facial image; encrypting the identity encoding information of the first facial image to obtain target identity encoding information; and generating the second facial image through the generation module based on the attribute encoding information and the target identity encoding information of the first facial image.

[0147] Optionally, in this embodiment of the application, the processing unit 204 is specifically used to: obtain the coordinate information of each pixel of the first facial image; replace the pixels of the first facial image with the pixels of the second facial image according to the coordinate information of each pixel of the first facial image; and smooth the pixels of the second facial image and the pixels of the original image to obtain the target image.

[0148] In the image protection device provided in this application embodiment, a first facial image is first obtained from the original image; then, the first facial image is parsed using a facial network model to obtain edge information of the first facial image, which is used to characterize the contours of each facial part in the first facial image; next, the first facial image and the edge information of the first facial image are image encoded using a variational autoencoder to obtain a second facial image; finally, the first facial image in the original image is updated to the second facial image to obtain the target image. Since the above image parsing can obtain the contours of each facial part in the facial image, when the variational autoencoder is used for subsequent encoding, only the content of each facial part in the facial image is encoded, while the contours of each facial part remain unchanged. Therefore, it is ensured that the generated second facial image, while scrambling the image content of the first facial image, also has the attribute features of the first facial image, so that the final target image protects user privacy and is identifiable.

[0149] When implemented in hardware, the acquisition unit 201, parsing unit 202, encoding unit 203, and processing unit 204 in this embodiment can be integrated onto a processor. Specific implementation methods are as follows: Figure 7 As shown.

[0150] Figure 7 A schematic diagram of another possible structure of the image protection device involved in the above embodiments is shown. This image protection device includes a processor 302 and a communication interface 303. The processor 302 is used to control and manage the operation of the image protection device, for example, executing the steps performed by the acquisition unit 201, parsing unit 202, encoding unit 203, and processing unit 204, and / or performing other processes of the technology described herein. The communication interface 303 is used to support communication between the image protection device and other network entities. The image protection device may also include a memory 301 and a bus 304, the memory 301 being used to store the program code and data of the image protection device.

[0151] The memory 301 may be a memory in an image protection device, and the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0152] The processor 302 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0153] Bus 304 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 304 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0154] Figure 8 This is a schematic diagram of the structure of chip 170 provided in an embodiment of this application. Chip 170 includes one or more (including two) processors 1710 and communication interfaces 1730.

[0155] Optionally, the chip 170 also includes a memory 1740, which may include read-only memory and random access memory, and provides operation instructions and data to the processor 1710. A portion of the memory 1740 may also include non-volatile random access memory (NVRAM).

[0156] In some implementations, memory 1740 stores elements such as execution modules or data structures, or subsets thereof, or extended sets thereof.

[0157] In this embodiment of the application, the corresponding operation is executed by calling the operation instructions stored in the memory 1740 (the operation instructions can be stored in the operating system).

[0158] The processor 1710 described above can implement or execute various exemplary logic blocks, units, and circuits described in conjunction with the disclosure of this application. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, units, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0159] The memory 1740 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include combinations of the above types of memory.

[0160] The Bus 1720 can be an Extended Industry Standard Architecture (EISA) bus, etc. The Bus 1720 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0161] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0162] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the image protection method described in the above method embodiments.

[0163] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the image protection method in the method flow shown in the above method embodiments.

[0164] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections having one or more wires; portable computer disks; hard disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); registers; hard disks; optical fibers; portable compact disc read-only memory (CD-ROM); optical storage devices; magnetic storage devices; or any suitable combination thereof; or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In the embodiments of this application, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0165] Embodiments of the present invention provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the image protection method described in the above-described method embodiments.

[0166] Since the image protection device, computer-readable storage medium, and computer program product in the embodiments of the present invention can be applied to the above method, the technical effects obtained can also be referred to the above method embodiments. The embodiments of the present invention will not be described again here.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0168] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0170] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image protection method characterized by, The method comprises: obtaining a first face image from an original image; performing image analysis on the first face image by a face network model to obtain edge information of the first face image, the edge information being used to represent the contour of each face part in the first face image; performing image encoding on the first face image and the edge information of the first face image by a variational autoencoder to obtain a second face image, the variational autoencoder comprising an encoding module and a generating module; updating the first face image in the original image to the second face image to obtain a target image; wherein the performing image analysis on the first face image by the face network model to obtain the edge information of the first face image comprises: performing image segmentation on the first face image by the face network model to obtain N region blocks, each region block corresponding to a face part in the first face image, N being a positive integer; obtaining the edge information of each region block in the N region blocks and determining the edge information of the N region blocks as the edge information of the first face image; the performing image encoding on the first face image and the edge information of the first face image by the variational autoencoder to obtain the second face image comprises: performing image encoding on the first face image by the encoding module to obtain semantic feature information, the semantic feature information being used to represent the image content and image features of each face part in the first face image; fusing the semantic feature information and the edge information of the first face image to obtain identity encoding information and attribute encoding information of the first face image, the identity encoding information being used to represent the encoding result of the image content of each face part in the first face image, and the attribute encoding information being used to represent the encoding result of the image features of each face part in the first face image; encrypting the identity encoding information of the first face image to obtain target identity encoding information; generating the second face image according to the attribute encoding information of the first face image and the target identity encoding information by the generating module.

2. The method of claim 1, wherein, The obtaining a first face image from an original image comprises: performing image detection on the original image to obtain coordinate information of each pixel point in the original image; segmenting the original image according to the coordinate information of each pixel point to obtain the first face image.

3. The method according to claim 1 or 2, characterized in that, The updating the first face image in the original image to the second face image to obtain a target image comprises: obtaining coordinate information of each pixel point of the first face image; replacing the pixel points of the first face image with the pixel points of the second face image according to the coordinate information of each pixel point of the first face image; performing smoothing processing on the pixel points of the second face image and the pixel points of the original image to obtain the target image.

4. An image protection apparatus characterized by comprising: The device comprises an obtaining unit, an analyzing unit, an encoding unit and a processing unit. The acquisition unit is configured to acquire a first face image from the original image; The analysis unit is configured to perform image analysis on the first face image acquired by the acquisition unit by using a face network model to obtain edge information of the first face image, the edge information being used to represent outlines of each face part in the first face image; The encoding unit is configured to perform image encoding on the first face image acquired by the acquisition unit and the edge information of the first face image obtained by the analysis unit by using a variational autoencoder to obtain a second face image, the variational autoencoder including an encoding module and a generation module; The processing unit is configured to update the first face image in the original image to the second face image to obtain a target image; The analysis unit is specifically configured to: perform image segmentation on the first face image by using the face network model to obtain N region blocks, each region block corresponding to one face part in the first face image, N being a positive integer; acquire edge information of each region block in the N region blocks, and determine the edge information of the N region blocks as the edge information of the first face image; The encoding unit is specifically configured to: perform image encoding on the first face image by using the encoding module to obtain semantic feature information, the semantic feature information being used to represent image content and image features of each face part in the first face image; fuse the semantic feature information and the edge information of the first face image to obtain identity encoding information and attribute encoding information of the first face image, the identity encoding information being used to represent encoding results of the image content of each face part in the first face image, and the attribute encoding information being used to represent encoding results of the image features of each face part in the first face image; encrypt the identity encoding information of the first face image to obtain target identity encoding information; generate the second face image according to the attribute encoding information of the first face image and the target identity encoding information by using the generation module.

5. The apparatus of claim 4, wherein: The acquisition unit is specifically configured to: perform image detection on the original image to obtain coordinate information of each pixel point in the original image; segment the original image according to the coordinate information of each pixel point to obtain the first face image.

6. The apparatus of claim 4 or 5, wherein: The processing unit is specifically configured to: acquire coordinate information of each pixel point of the first face image; replace the pixel points of the first face image with pixel points of the second face image according to the coordinate information of each pixel point of the first face image; perform smoothing processing on the pixel points of the second face image and the pixel points of the original image to obtain the target image.

7. An image protection apparatus characterized by comprising: including: a processor and a communication interface; the communication interface and the processor are coupled, and the processor is configured to run computer programs or instructions to implement the image protection method as claimed in any one of claims 1-3.

8. A computer-readable storage medium having stored therein instructions, the computer-readable storage medium comprising: When a computer executes the instructions, the computer executes the image protection method as claimed in any one of claims 1-3.

Images