Video face privacy protection method based on residual structure encoder-decoder architecture

Abstract

The present invention discloses a video face privacy protection method based on a residual structure encoder-decoder architecture, and relates to the technical field of video privacy protection. The method comprises: collecting face video data shot by a camera and processing it into first picture data, extracting face picture data in the first picture data; blurring the face picture data to obtain blurred face picture data; training a face and blurred face binary classification model, using the binary classification model to identify the face picture data and the blurred face picture data; training a residual structure complex encoder model, a residual structure complex decoder model and a password model; the present invention can achieve a high degree of privacy protection without reducing the video quality, and can highly restore the privacy-protected video after obtaining password authorization. At the same time, the operation of the method is very simple, and only the trained model needs to be connected to the device where the video needs to be hidden.

Description

Technical Field

[0001] The present invention relates to the technical field of video privacy protection, and in particular to a video face privacy protection method based on a residual structure encoder-decoder architecture. Background Art

[0002] With the continuous development and popularization of surveillance technology, surveillance cameras have become an important part of the security systems of many public places, commercial buildings, communities and homes. The use of surveillance cameras can provide real-time monitoring, record events, prevent crimes and other functions, which plays a vital role in maintaining social security and providing evidence.

[0003] However, with the popularity of surveillance cameras, issues related to personal privacy protection have become increasingly prominent. Surveillance cameras may collect a large amount of personal information, such as personal identity, facial features, behavior, etc., which may be abused or used without authorization. In addition, the images and videos captured by surveillance cameras within the monitoring range may contain some sensitive information, such as bank card numbers, passwords, device features, etc. If this information falls into the hands of criminals, it may lead to risks such as identity theft and property loss. Summary of the invention

[0004] (I) Purpose of the invention

[0005] The purpose of the present invention is to provide a video face privacy protection method based on a residual structure encoder-decoder architecture to solve the problem that surveillance cameras in the prior art cannot effectively protect privacy and infringe on the privacy of others during surveillance shooting.

[0006] (II) Technical solution

[0007] To solve the above problems, the present invention provides a video face privacy protection method based on a residual structure encoder-decoder architecture, comprising:

[0008] Step 1: collect face video data captured by a camera and process it into first picture data, and extract face picture data from the first picture data;

[0009] Step 2, blurring the face image data to obtain blurred face image data;

[0010] Step 3: training a binary classification model for human faces and blurred human faces, and using the binary classification model to identify the human face image data and the blurred human face image data;

[0011] Step 4: training a residual structure complex encoder model, a residual structure complex decoder model and a password model;

[0012] Step 5: input the face picture data into the residual structure complex encoder model to obtain a result picture, put the result picture back into the first picture data, and then convert the first picture data into blurred face video data;

[0013] Step 6: Convert the blurred face video data into second picture data and extract the blurred face picture data therein, add the blurred face picture data and the password data obtained using the password model and put them into the residual structure complex decoder to obtain the restored face picture data, put the restored face picture data back into the second picture data, and then restore the second picture data to face video data.

[0014] Optionally, step 1, collecting face video data captured by a camera and processing it into first picture data, and extracting face picture data from the first picture data includes:

[0015] Step 1.1, write Python code to process the face video data frame by frame into image data;

[0016] Step 1.2, using the face detection method in OpenCV to detect the face picture data in the image data;

[0017] Step 1.3, write Python code to extract the detected face image data;

[0018] Step 1.4, putting the face image data into the created first folder, and setting the RGB value of the face image position in the image data to (0, 0, 0);

[0019] Step 1.5, write Python code to create a txt file, and record the location information of the detected face image data in the txt file.

[0020] Optionally, step 2, blurring the face image data to obtain blurred face image data, comprises:

[0021] Step 2.1, using the resize function of the Python PIL module to compress the face image data;

[0022] Step 2.2: The nearest neighbor interpolation algorithm is selected as the interpolation method, so as to take the gray value of the nearest neighbor point among the four neighboring pixels around the sampling point in the two-dimensional image as the gray value of the point;

[0023] Step 2.3, again using the resize function of the Python PIL module to restore the compressed face image data to the blurred face image data, and putting the blurred face image data into the created second folder, wherein the interpolation method selects the bicubic interpolation algorithm.

[0024] Optionally, the face and blurred face binary classification model mainly consists of four convolutional layers, four batch normalization layers, four maximum pooling layers, one Flatten layer and three linear layers.

[0025] Optionally, step 3, training a binary classification model of human face and blurred human face, and using the binary classification model to identify the human face image data and the blurred human face image data comprises:

[0026] Step 3.1, construct the face and fuzzy face binary classification model structure;

[0027] Step 3.2, write Python code to inherit the Dataset class in the torch.utils.data module to read the data in the first folder and the second folder in the form of {"img": image data, "label": label}, use the transforms.Resize function to process the first image data into set pixels during the reading process, and then use the transforms.Normalize function to normalize the first image data;

[0028] Step 3.3, merging the data read from the first folder and the second folder together;

[0029] Step 3.4, write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input it into the face and blurred face binary classification model in batches.

[0030] Optionally, in step 3.4, write a python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input them in batches into the face and fuzzy face binary classification model, and then include:

[0031] Step 3.5: The loss function of the face and fuzzy face binary classification model is CrossEntropyLoss loss function. The formula of CrossEntropyLoss loss function is as follows:

[0032] H(p,q)=-Σ x (p(x)log q(x))

[0033] Among them, p is the expected output, q is the actual output, and x is the sample.

[0034] Optionally, step 4, training the residual structure complex encoder model, the residual structure complex decoder model and the password model includes:

[0035] Step 4.1, building the residual structure complex encoder model, the residual structure complex decoder model and the password model, wherein the residual structure complex encoder model and the residual structure complex decoder model are built using a network layer, and a residual neural network is used in this process;

[0036] Step 4.2, write Python code to inherit the Dataset class in the torch.utils.data module to read the first folder in the form of {"img": image data, "label": label}. During the reading process, first use the transforms.Resize function to process the image data into set pixels, and then use the transforms.Normalize function to normalize it;

[0037] Step 4.3, write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input them in batches into the residual structure complex encoder model to obtain the result data, add the result data and the password data to obtain the intermediate data, and input the intermediate data into the residual structure complex decoder model;

[0038] Step 4.4, simultaneously training the residual structure complex encoder model, the residual structure complex decoder model and the password model, with multiple loss functions;

[0039] Step 4.5. Select Adam as the optimizer and use the face image data to train the residual structure complex encoder model, the residual structure complex decoder model and the password model until the total loss function value of the residual structure complex encoder model, the residual structure complex decoder model and the password model reaches a smaller value, and the total loss function value is equal to the sum of multiple loss function values.

[0040] (III) Beneficial effects

[0041] The above technical solution of the present invention has the following beneficial technical effects:

[0042] In the present invention, on the one hand, only the face in the surveillance video captured by the surveillance camera is blurred, the face video data captured by the camera is processed into the first picture data, and the face in the first picture data is blurred, and the residual structure complex encoder model, the residual structure complex decoder model and the password model are used to convert the first picture data into blurred face video data, so that the personal privacy in the surveillance video can be well protected under normal circumstances. More importantly, in the present invention, in order to prevent the blurred video from being illegally restored, the present invention sets a double insurance. Since the present invention adds a password model during the training process, when the blurred face video data and the residual structure complex decoder are owned by others, but the other party does not have the password, the blurred face video data cannot be restored. When the blurred face video data and the password are owned by others, but the other party does not have the residual structure complex decoder, the blurred face video data cannot be restored either.

[0043] Therefore, the method of the present invention can not only achieve a high degree of privacy protection without reducing the quality of the video, but also highly restore the privacy-protected video after obtaining password authorization. At the same time, the operation of the method is very simple, and only requires connecting the trained model to the device where the video needs to be hidden. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic flow chart of the main control process of the video face privacy protection method of the present invention;

[0045] Figure 2 is a schematic flow chart of the overall framework of the video face privacy protection method of the present invention;

[0046] Figure 3 is a schematic flow chart of the face recognition process in the video face privacy protection method of the present invention;

[0047] Figure 4 It is a schematic flow chart of the cryptographic model structure in the video face privacy protection method of the present invention;

[0048] Figure 5 It is a schematic flow chart of the binary classification model structure of face and fuzzy face in the video face privacy protection method of the present invention;

[0049] Figure 6 It is a schematic flow chart of the residual structure complex encoder model structure in the video face privacy protection method of the present invention;

[0050] Figure 7 It is a schematic flow chart of the residual structure complex decoder model structure in the video face privacy protection method of the present invention;

[0051] Figure 8 It is an overall schematic flow chart of the effects of each step in the video face privacy protection method of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0053] The accompanying drawings show schematic diagrams of layer structures according to embodiments of the present invention. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clarity. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0054] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0056] The present invention will be described in more detail below with reference to the accompanying drawings. In each of the accompanying drawings, the same elements are represented by similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not drawn to scale.

[0057] Existing components that are not related to the improvement points of the present invention will be briefly introduced or not introduced at all, while the focus will be on introducing the components that are improved compared to the prior art.

[0058] See also Figures 1 to 8 The present invention provides a video face privacy protection method based on a residual structure encoder-decoder architecture, comprising:

[0059] Step 1: collect face video data captured by a camera and process it into first picture data, and extract face picture data from the first picture data;

[0060] Step 2, blurring the face image data to obtain blurred face image data;

[0061] Step 3: training a binary classification model for human faces and blurred human faces, and using the binary classification model to identify the human face image data and the blurred human face image data;

[0062] Step 4: training a residual structure complex encoder model, a residual structure complex decoder model and a password model;

[0063] Step 5: input the face picture data into the residual structure complex encoder model to obtain a result picture, put the result picture back into the first picture data, and then convert the first picture data into blurred face video data;

[0064] Step 6: Convert the blurred face video data into second picture data and extract the blurred face picture data therein, add the blurred face picture data and the password data obtained using the password model and put them into the residual structure complex decoder to obtain the restored face picture data, put the restored face picture data back into the second picture data, and then restore the second picture data to face video data.

[0065] It should be noted that the camera in this embodiment can be a surveillance camera in a public place, such as a store, supermarket, factory, cinema, community or bus; in addition, the camera here can also be a camera in a driving recorder.

[0066] In the present invention, on the one hand, only the face in the surveillance video captured by the surveillance camera is blurred, the face video data captured by the camera is processed into the first picture data, and the face in the first picture data is blurred, and the residual structure complex encoder model, the residual structure complex decoder model and the password model are used to convert the first picture data into blurred face video data, so that the personal privacy in the surveillance video can be well protected under normal circumstances. More importantly, in the present invention, in order to prevent the blurred video from being illegally restored, the present invention sets a double insurance. Since the present invention adds a password model during the training process, when the blurred face video data and the residual structure complex decoder are owned by others, but the other party does not have the password, the blurred face video data cannot be restored. When the blurred face video data and the password are owned by others, but the other party does not have the residual structure complex decoder, the blurred face video data cannot be restored either.

[0067] Therefore, the method of the present invention can not only achieve a high degree of privacy protection without reducing the quality of the video, but also highly restore the privacy-protected video after obtaining password authorization. At the same time, the operation of the method is very simple, and only requires connecting the trained model to the device where the video needs to be hidden.

[0068] For the convenience of description, in this embodiment, a surveillance camera in a supermarket is taken as an example for explanation.

[0069] See also Figures 1 to 8 Further, step 1, collecting face video data captured by a camera and processing it into first picture data, extracting face picture data from the first picture data includes:

[0070] Step 1.1, write Python code to process the face video data frame by frame into image data;

[0071] Step 1.2, using the face detection method in OpenCV to detect the face picture data in the image data;

[0072] Step 1.3, write Python code to extract the detected face image data;

[0073] Step 1.4, putting the face image data into the created first folder, and setting the RGB value of the face image position in the image data to (0, 0, 0);

[0074] Step 1.5, write Python code to create a txt file, and record the location information of the detected face image data in the txt file.

[0075] That is, step 1, collecting face video data taken by a camera and processing it into first picture data, extracting face picture data from the first picture data may be, collecting face video data taken by a supermarket surveillance camera, processing the video data into picture data, extracting face pictures appearing in each picture data, and recording the extraction position of the face picture in the picture data in a txt file;

[0076] Prioritize, collect face video data taken by supermarket surveillance cameras, process the video data into image data, extract face images appearing in each image data, and record the extraction position of the face image in the image data in a txt file. Specifically, collect face video data taken by supermarket surveillance cameras, write Python code to process the video data frame by frame into image data, use the face detection method in OpenCV to detect faces in the image data, write Python code to extract the detected face image data, create a folder named 1, put the face image data into folder 1, and set the RGB value at the face position of the image data after extracting the face image data to (0, 0, 0), write Python code to create a txt file, and record the position information of the detected face image data in the txt file.

[0077] See also Figures 1 to 8Further, step 2, blurring the face image data to obtain blurred face image data includes:

[0078] Step 2.1, using the resize function of the Python PIL module to compress the face image data;

[0079] Step 2.2: The nearest neighbor interpolation algorithm is selected as the interpolation method, so as to take the gray value of the nearest neighbor point among the four neighboring pixels around the sampling point in the two-dimensional image as the gray value of the point;

[0080] Step 2.3, again using the resize function of the Python PIL module to restore the compressed face image data to the blurred face image data, and putting the blurred face image data into the created second folder, wherein the interpolation method selects the bicubic interpolation algorithm.

[0081] That is, the extracted face image data is fuzzy processed to obtain fuzzy face image data;

[0082] Preferably, the extracted face image data is blurred to obtain blurred face image data. Specifically, the face image data is processed to one tenth of the original size using the resize function of the Python PIL module, and the nearest neighbor interpolation algorithm is selected as the interpolation method.

[0083] This method takes the grayscale value of the nearest neighboring point among the four adjacent pixels around the sampling point in a two-dimensional image as the grayscale value of the point. It is a fast but low-precision image pixel simulation method, and obtains face image data of one-tenth size. The resize function of the Python PIL module is used again to restore the face image data of one-tenth size to the original size. The interpolation method selects the two-cubic interpolation algorithm. This method not only takes into account the influence of the grayscale values ​​of the four direct neighboring points, but also takes into account the influence of the grayscale value change rate between each neighboring point. It uses the grayscale values ​​of pixels in a larger neighborhood around the sampling point for cubic interpolation, which is an improvement on the two-linear interpolation algorithm. The calculation accuracy is very high, and the image pixel loss after processing is small. At this time, the face image data restored to the original size is the blurred face image data. Create a folder named 0 and put the blurred face image data into folder 0.

[0084] See also Figures 1 to 8 ,Furthermore, the face and blurred face binary classification model is mainly composed of four convolutional layers, four batch normalization layers, four maximum pooling layers, one Flatten layer and three linear layers.

[0085] See also Figures 1 to 8Further, step 3, training a binary classification model of human face and fuzzy human face, and using the binary classification model to identify the human face image data and the fuzzy human face image data includes:

[0086] Step 3.1, construct the face and fuzzy face binary classification model structure;

[0087] Step 3.2, write Python code to inherit the Dataset class in the torch.utils.data module to read the data in the first folder and the second folder in the form of {"img": image data, "label": label}, use the transforms.Resize function to process the first image data into set pixels during the reading process, and then use the transforms.Normalize function to normalize the first image data;

[0088] Step 3.3, merging the data read from the first folder and the second folder together;

[0089] Step 3.4, write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input it into the face and blurred face binary classification model in batches.

[0090] Step 3.5: The loss function of the face and fuzzy face binary classification model is CrossEntropyLoss loss function. The formula of CrossEntropyLoss loss function is as follows:

[0091] H(p,q)=-Σ x (p(x)log q(x))

[0092] Among them, p is the expected output, q is the actual output, and x is the sample.

[0093] In other words, a face and fuzzy face binary classification model is trained to determine whether the image is a face image or a fuzzy face image. Specifically, the face and fuzzy face binary classification model consists of four convolutional layers, four batch normalization layers, four maximum pooling layers, one Flatten layer and three linear layers. The construction and training of the binary classification model can be done through the following steps:

[0094] Step 3.1: Face and fuzzy face binary classification model structure:

[0095] Convolution layer: 32 channels, 5*5 convolution kernel, batch normalization layer, maximum pooling layer,

[0096] Convolution layer: 64 channels, 5*5 convolution kernel, batch normalization layer, maximum pooling layer,

[0097] Convolution layer: 128 channels, 5*5 convolution kernel, batch normalization layer, maximum pooling layer,

[0098] Convolution layer: 256 channels, 5*5 convolution kernel, batch normalization layer, maximum pooling layer,

[0099] Flatten layer, linear layer, linear layer, linear layer;

[0100] Step 3.2: Write Python code to inherit the Dataset class in the torch.utils.data module and read the folder named 1 and the folder named 0 in the form of {"img": picture data, "label": label}, for example, {"img": face picture, "label": 1}, {"img": blurred face picture, "label": 0}. During the reading process, the transforms.Resize function is first used to process the picture data into a size of 256×256 pixels, and then the transforms.Normalize function is used to standardize the picture data, that is, the mean is 0 and the standard deviation is 1;

[0101] Step 3.3: Merge the data read from the two folders together;

[0102] Step 3.4: Write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input it into the binary classification model in batches. The DataLoader parameters are set to batch_size = 16, shuffle = True, num_workers = 6, drop_last = True;

[0103] Step 3.5: The loss function of the face and fuzzy face binary classification model selects the CrossEntropyLoss loss function. The CrossEntropyLoss loss function formula is as follows:

[0104] H(p,q)=-Σ x (p(x)log q(x))

[0105] Among them: p is the expected output, q is the actual output, and x is the sample.

[0106] The optimizer selects SGD. Since this classification problem is relatively simple, a good two-classification model can be obtained by training for more than ten epochs.

[0107] See also Figures 1 to 8, further, step 4, training the residual structure complex encoder model, the residual structure complex decoder model and the password model includes:

[0108] Step 4.1, building the residual structure complex encoder model, the residual structure complex decoder model and the password model, wherein the residual structure complex encoder model and the residual structure complex decoder model are built using a network layer, and a residual neural network is used in this process;

[0109] Step 4.2, write Python code to inherit the Dataset class in the torch.utils.data module to read the first folder in the form of {"img": image data, "label": label}. During the reading process, first use the transforms.Resize function to process the image data into set pixels, and then use the transforms.Normalize function to normalize it;

[0110] Step 4.3, write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input them in batches into the residual structure complex encoder model to obtain the result data, add the result data and the password data to obtain the intermediate data, and input the intermediate data into the residual structure complex decoder model;

[0111] Step 4.4, simultaneously training the residual structure complex encoder model, the residual structure complex decoder model and the password model, with multiple loss functions;

[0112] Step 4.5. Select Adam as the optimizer and use the face image data to train the residual structure complex encoder model, the residual structure complex decoder model and the password model until the total loss function value of the residual structure complex encoder model, the residual structure complex decoder model and the password model reaches a smaller value, and the total loss function value is equal to the sum of multiple loss function values.

[0113] At the same time, a residual complex structure encoder model, a residual complex structure decoder model and a password model are trained. The face image is input into the residual complex structure encoder network to obtain the result image. The closer the result image is to the blurred face image, the better. A nine-digit password is input into the password model to obtain a password data of the same size as the result image. The result image and the password data are added to obtain the intermediate data. The intermediate data is input into the residual complex structure decoder network to obtain the restored image. The closer the restored image is to the face image, the better.

[0114] Preferably, a residual complex structure encoder model, a residual complex structure decoder model and a password model are trained simultaneously. A face picture is input into the residual complex structure encoder network to obtain a result picture. The closer the result picture is to the blurred face picture, the better. A nine-digit password is input into the password model to obtain password data of the same size as the result picture. The result picture and the password data are added to obtain intermediate data. The intermediate data is input into the residual complex structure decoder network to obtain a restored picture. The closer the restored picture is to the face picture, the better. Specifically, the residual complex structure encoder model and the residual complex structure decoder model are composed of 21 convolutional layers, 21 LeakReLU layers and 21 instance normalization layers (InstanceNorm2d). The password model consists of three linear layers. The construction and training of the residual complex structure encoder model, the residual complex structure decoder model and the password model can be carried out through the following steps:

[0115] Step 4.1: Build a residual structure complex encoder model, a residual structure complex decoder model and a password model. When using network layers to build the residual structure complex encoder model and the residual structure complex decoder model, the present invention uses a residual neural network, which greatly eliminates the difficulty in training a neural network model with too large a depth.

[0116] Residual structure complex encoder model structure:

[0117] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0118] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0119] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0120] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0121] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0122] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0123] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0124] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0125] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0126] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0127] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0128] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0129] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0130] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0131] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0132] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0133] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0134] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0135] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0136] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0137] Convolutional layer: 3 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer;

[0138] Residual structure complex decoder model structure:

[0139] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0140] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0141] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0142] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0143] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0144] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0145] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0146] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0147] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0148] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0149] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0150] Convolution layer: 256 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0151] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0152] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0153] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0154] Convolution layer: 128 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0155] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0156] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0157] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0158] Convolution layer: 64 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer,

[0159] Convolutional layer: 3 channels, 3*3 convolution kernel, LeakReLU layer, instance normalization layer;

[0160] Cryptographic model structure: linear layer, linear layer, linear layer;

[0161] Step 4.2: Write Python code to inherit the Dataset class in the torch.utils.data module and read the folder named 1 in the form of {"img": picture data, "label": label}, for example, {"img": face picture, "label": 1}. During the reading process, the transforms.Resize function is first used to process the picture data into a size of 256×256 pixels, and then the transforms.Normalize function is used to standardize the picture data, that is, the mean is 0 and the standard deviation is 1;

[0162] Step 4.3: Write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input them in batches into the residual structure complex encoder model to obtain the result data, add the result data and the password data to obtain the intermediate data, and input the intermediate data into the residual structure complex decoder model;

[0163] The DataLoader parameters are set to batch_size=16, shuffle=True, num_workers=6, drop_last=True.

[0164] Step 4.4: Train a residual structure complex encoder model, a residual structure complex decoder model and a password model at the same time. There are four loss functions. The first loss function is to use the MSELoss loss function between the result image and the corresponding blurred face image. The MSELoss loss function is as follows:

[0165]

[0166]

[0167] Where: N is the number of samples, l n is the loss value of the nth sample, x n is the neural network output of the nth sample, y n is the true score of the nth sample, and T is the transpose.

[0168] The second loss function is to use the MSELoss loss function between the restored image and the face image. The MSELoss loss function is as follows:

[0169]

[0170]

[0171] Where: N is the number of samples, l n is the loss value of the nth sample, x n is the neural network output of the nth sample, y n is the true score of the nth sample, and T is the transpose.

[0172] The third loss function is to put the intermediate data into the binary classification model to obtain the classification label, and use the CrossEntropyLoss loss function between the classification label and 0. The formula of the CrossEntropyLoss loss function is as follows:

[0173] H(p,q)=-Σ X (p(x)log q(x))

[0174] Among them: p is the expected output, q is the actual output, and x is the sample.

[0175] The fourth loss function is to use the |MSELoss-0.05| loss function between the result image and the intermediate image. The |MSELoss-0.05| loss function is as follows:

[0176]

[0177]

[0178]

[0179] Where: N is the number of samples, l n is the loss value of the nth sample, x n is the neural network output of the nth sample, y n is the true score of the nth sample, and T is the transpose.

[0180] Step 4.5: Select Adam as the optimizer and use the face image data to train the model until the total loss function value of the model reaches a smaller value, which is equal to the sum of the four loss function values.

[0181] Step 5: input the face picture data into the residual structure complex encoder model to obtain a result picture, put the result picture back into the first picture data, and then convert the first picture data into blurred face video data.

[0182] That is to say, the collected face image data is input into the residual structure complex encoder model to obtain the result image, the result image is put back into the image data, and then the image data is converted into blurred face video data, so that the face appearing in the original video is effectively protected;

[0183] Prioritizing, the collected face image data is input into the residual structure complex encoder model to obtain a result image, the result image is put back into the image data, and the image data is converted into blurred face video data, and the faces appearing in the original video are effectively protected. Specifically, the collected face image data is input into the residual structure complex encoder model to obtain a result image, the result image is put back into each image data according to the face extraction position recorded in the txt file, and the image data is converted into blurred face video data, and the faces appearing in the video are effectively protected.

[0184] Step 6: Convert the blurred face video data into second picture data and extract the blurred face picture data therein, add the blurred face picture data and the password data obtained using the password model and put them into the residual structure complex decoder to obtain the restored face picture data, put the restored face picture data back into the second picture data, and then restore the second picture data to face video data.

[0185] That is to say, the blurred face video data is converted into image data, the blurred face image is extracted, the blurred face image is added to the password data and put into the residual structure complex decoder to obtain the restored face image, the restored face image is put back into the image data, and the image data is converted into the restored face video data, and the face appearing in the blurred face video can be restored;

[0186] Preferably, the blurred face video data is converted into picture data, the blurred face picture is extracted, the blurred face picture is added to the password data and put into the residual structure complex decoder to obtain the restored face picture, the restored face picture is put back into the picture data, and the picture data is converted into the restored face video data, and the face appearing in the blurred face video can be restored. Specifically, the blurred face video data is converted into picture data, the blurred face picture is extracted according to the face extraction position recorded in the txt file, the blurred face picture is added to the password data and put into the residual structure complex decoder to obtain the restored face picture, the restored face picture is put back into the picture data according to the face extraction position recorded in the txt file, and the picture data is converted into the restored face video data, and the face appearing in the blurred face video can be restored.

[0187] In addition, the above steps 1 to 6 explain how to use the facial video data captured by the supermarket surveillance camera to train the model, and how to protect and restore the facial video data. If the facial video data captured by the supermarket surveillance camera has been used to train the residual structure complex encoder model, the password model and the residual structure complex decoder model, in actual use, the operator only needs to connect the supermarket surveillance camera to the residual structure complex encoder model to directly obtain the blurred facial video surveillance video. When an accident occurs and the supermarket surveillance video needs to be retrieved, the authorized operator can input the password and the blurred facial video into the password model and the residual structure complex decoder to directly obtain the restored face video.

[0188] Specifically, the video captured by the supermarket surveillance camera is converted into image data, face image data is extracted from the image data, the extraction position of the face image in the image data is recorded in a txt file, all face images are input into the residual structure complex encoder to obtain blurred face images, the blurred face images are put back into the image data according to the face extraction position recorded in the txt file, and the image data is converted into blurred face video data. The blurred face video data is finally saved by the supermarket surveillance camera. When an accident occurs and the supermarket surveillance video needs to be retrieved, the authorized operator can input the password into the password model to obtain the password data, convert the blurred face video data into image data, extract the blurred face image according to the face extraction position recorded in the txt file, add the blurred face image and the password data and put them into the residual structure complex decoder to obtain the restored face image, put the restored face image back into the image data according to the face extraction position recorded in the txt file, and then convert the image data into restored face video data to obtain the restored face video.

[0189] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present invention, and these various changes are applicable to the method of the present invention.

Claims

1. A video face privacy protection method based on a residual structure encoder-decoder architecture, characterized in that: include: Step 1: collecting face video data captured by a camera and processing it into first picture data, and extracting face picture data from the first picture data; Record the location information of the detected face image data in a txt file; Step 2, blurring the face image data to obtain blurred face image data; Step 3: training a binary classification model for human faces and blurred human faces, and using the binary classification model to identify the human face image data and the blurred human face image data; Step 4: training a residual structure complex encoder model, a residual structure complex decoder model and a password model; Step 5: input the face picture data into the residual structure complex encoder model to obtain a result picture, put the result picture back into the first picture data, and then convert the first picture data into blurred face video data; The result image is close to the blurred face image, and the result image is put back into the image data according to the face extraction position recorded in the txt file. Step 6. Convert the blurred face video data into second picture data and extract the blurred face picture data therein, input a 9-digit password into the password model, obtain password data of the same size as the result picture, add the blurred face picture data and the password data obtained using the password model and put them into the residual structure complex decoder, obtain the restored face picture data and put it back into the picture data according to the face extraction position recorded in the txt file, put the restored face picture data back into the second picture data, and then restore the second picture data to face video data.

2. The video face privacy protection method according to claim 1, characterized in that: Step 1: collecting face video data captured by a camera and processing it into first picture data, extracting face picture data from the first picture data includes: Step 1.1, write Python code to process the face video data frame by frame into image data; Step 1.2, using the face detection method in OpenCV to detect the face picture data in the image data; Step 1.3, write Python code to extract the detected face image data; Step 1.4, putting the face image data into the created first folder, and setting the RGB value of the face image position in the image data to (0, 0, 0); Step 1.5: Write Python code to create a txt file, and record the location information of the detected face image data in the txt file.

3. The video face privacy protection method according to claim 2, characterized in that: Step 2: blurring the face image data to obtain blurred face image data, including: Step 2.1, using the resize function of the Python PIL module to compress the face image data; Step 2.2: Select the nearest neighbor interpolation algorithm as the interpolation method, and take the gray value of the nearest neighbor point among the four neighboring pixels around the sampling point in the two-dimensional image as the gray value of the point; Step 2.3: Use the resize function of the Python PIL module again to restore the compressed face image data to the blurred face image data, and put the blurred face image data into the created second folder, wherein the interpolation method selects the bicubic interpolation algorithm.

4. The video face privacy protection method according to claim 3, characterized in that: The face and blurred face binary classification model mainly consists of four convolutional layers, four batch normalization layers, four maximum pooling layers, one Flatten layer and three linear layers.

5. The video face privacy protection method according to claim 4, characterized in that: Step 3: training a binary classification model of human face and blurred human face, and using the binary classification model to identify the human face picture data and the blurred human face picture data comprises: Step 3.1, construct the face and fuzzy face binary classification model structure; Step 3.2, write Python code to inherit the Dataset class in the torch.utils.data module, read the data in the first folder and the second folder in the form of {"img": image data, "label": label}, use the transforms.Resize function to process the first image data into set pixels during the reading process, and then use the transforms.Normalize function to normalize the first image data; Step 3.3, merging the data read from the first folder and the second folder together; Step 3.4, write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input them into the face and blurred face binary classification model in batches.

6. The video face privacy protection method according to claim 5, characterized in that: Step 3.4, write Python code to inherit the DataLoader class in the torch.utils.data module to randomly shuffle the merged data and input it into the face and fuzzy face binary classification model in batches, including: Step 3.5: The loss function of the face and fuzzy face binary classification model selects the CrossEntropyLoss loss function, and the CrossEntropyLoss loss function formula is as follows: Wherein, p is the expected output, q is the actual output, and x is the sample.

7. The video face privacy protection method according to claim 5, characterized in that: Step 4: Training the residual structure complex encoder model, the residual structure complex decoder model and the password model includes: Step 4.1, building the residual structure complex encoder model, the residual structure complex decoder model and the password model, wherein the residual structure complex encoder model and the residual structure complex decoder model are built using a residual neural network; Step 4.2, write Python code to inherit the Dataset class in the torch.utils.data module, read the first folder in the form of {"img": image data, "label": label}, and use the transforms.Resize function to process the image data into set pixels during the reading process, and then use the transforms.Normalize function to normalize it; Step 4.3, write Python code to inherit the DataLoader class in the torch.utils.data module, randomly shuffle the merged data, and input the residual structure complex encoder model in batches to obtain the result data, input a 9-digit password into the password model, obtain a password data of the same size as the result image, add the result data and the password data to obtain the intermediate data, and input the intermediate data into the residual structure complex decoder model; Step 4.4, simultaneously training the residual structure complex encoder model, the residual structure complex decoder model and the password model, with multiple loss functions; Step 4.

5. Select Adam as the optimizer and use the face image data to train the residual structure complex encoder model, the residual structure complex decoder model and the password model until the total loss function value of the residual structure complex encoder model, the residual structure complex decoder model and the password model reaches a minimum value, and the total loss function value is equal to the sum of multiple loss function values.

Images