A method for coding a specific target person in a video

Abstract

The present invention belongs to the field of image processing technology, and in particular, relates to a method for coding a specific target person in a video. The method comprises: (1) constructing a target person face database; (2) extracting video frames from a video stream, performing face detection, position marking, and face correction using a trained model; (3) extracting feature vectors from the corrected face image using an improved residual network, comparing the feature vectors with the feature vectors of the target person in the face database, and if the similarity is greater than a preset threshold, marking the face image as a target person to be coded; (4) coding the marked target person; (5) repeating operations (2) to (4) for each frame of the video, and synthesizing the video stream frame by frame for output. The present invention improves the residual network for generating facial feature vectors, and by changing the first step of convolution, better preserves image details, and significantly improves the recognition accuracy of small-sized faces in the image.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method for coding a specific target person in a video. Background Art

[0002] Face recognition is a biometric technology that distinguishes targets based on facial features. It distinguishes targets by detecting the face position, analyzing the face key points and generating face feature vectors.

[0003] With the rapid development of short videos and variety shows, it is sometimes necessary to continuously blur certain objects in the video for privacy or other reasons. The current approach to this problem is to use video editing software to manually blur certain target people, or to automatically blur all faces. This method has the problems of cumbersome steps, long time consumption, and inability to accurately blur. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method for mosaic processing of specific targets in a video, which utilizes face recognition technology and mosaic technology to protect user privacy and improves the recognition accuracy of small-sized faces to a certain extent.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a method for coding a specific target person in a video, comprising:

[0006] (1) Build a target person face database;

[0007] (2) Extract video frames from the video stream, perform face detection, position marking, and perform face correction using the trained model;

[0008] (3) Using the improved residual network to extract feature vectors from the corrected face image, the feature vectors are compared with the feature vectors of the target person in the face database. If the similarity is greater than a preset threshold, the face image is marked as the target person to be coded;

[0009] (4) Coding the marked target person;

[0010] (5) After repeating operations (2) to (4) for each frame of the video, the video stream is synthesized frame by frame and output.

[0011] Furthermore, the method for constructing the target person face database in step (1) includes:

[0012] Collect sample images of the target person;

[0013] Use the trained model to detect 68 feature points of the sample image and use the feature points to correct the image;

[0014] Extract 128D feature vector from the rectified image through the improved residual network;

[0015] The average value of the feature vectors generated by all sample images is calculated and stored in the database as the feature vector of the target person.

[0016] Furthermore, in the step (2), the input video frame is grayed out, and the processed image is subjected to face detection using a hierarchical detection network to determine the position of the face sample in the image.

[0017] Furthermore, the hierarchical detection network is a YOLO network comprising three detection modules, wherein the detection module is composed of a convolutional binary classifier and a regressor, and the step sizes of the three detection modules are respectively set to 32, 16, and 8 to detect faces of different sizes.

[0018] Furthermore, among the three detection modules, the receptive field is increased by Max-Pooling between the two modules for detecting larger face sizes to detect faces of larger sizes; the module for detecting smaller face sizes fuses the features obtained by the two convolutional layers by upsampling to detect faces of small sizes.

[0019] Furthermore, the trained model is a Dlib cascade regression tree model.

[0020] Furthermore, the improved residual network adds a feature enhancement layer at the front end of the network to enhance the expressiveness of the features and complete downsampling; the feature enhancement layer consists of a 7*7 convolution with 16 channels and a BasicBlock with 16 channels, followed by a BasicBlock with a step size of 2 and 32 channels.

[0021] Furthermore, the step (3) specifically includes:

[0022] Convert the 128D feature vector of the face of the person in the video frame into a column vector, denoted as b;

[0023] The facial feature vectors in the face database are used as row vectors and sorted by character labels to form a feature matrix A. n×128 , where n is the number of characters in the feature library;

[0024] Calculate vector similarity:

[0025]

[0026] Among them: diag() means taking the diagonal elements of the matrix to form a column vector; sim is an n×1 vector;

[0027] Get the most similar people:

[0028] When sim(j) is greater than the set threshold, it is considered to be a target person that needs to be coded.

[0029] Furthermore, the coding process includes selecting a given picture to cover the face or blur the face.

[0030] The method of the present invention has the following beneficial effects:

[0031] (1) Through face detection and recognition, a method for automatic video coding of specific target persons is provided, which meets people's requirements for high efficiency and accuracy of video coding processing.

[0032] (2) The present invention improves the face detection method by using a branched YOLO network to more accurately detect faces of various sizes in an image. At the same time, the residual network for generating facial feature vectors is improved by changing the first step of convolution to better preserve image details. The above improvements significantly improve the recognition accuracy of small-sized faces in images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the overall process of a method for coding a specific target person in a video disclosed in an embodiment of the present invention;

[0034] Figure 2 A detailed flow chart of a method for coding a specific target person in a video disclosed in an embodiment of the present invention;

[0035] Figure 3 An improved residual network structure diagram for coding a specific target person in a video disclosed in an embodiment of the present invention;

[0036] Figure 4 The effect diagram of using the method of the present invention to code a specific target person in a video, (a) before coding; (b) after coding. DETAILED DESCRIPTION

[0037] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0038] This embodiment provides a method for coding a specific target person in a video. Figure 1 and Figure 2 As shown in the figure, this method uses face recognition technology to protect the privacy of specific objects in the video, and to a certain extent improves the accuracy of small-size face recognition. The specific steps are described in detail as follows:

[0039] 1. Build a target person face database

[0040] For each target person, sample images are collected, and the trained model is used to detect the 68 feature points of the sample images. The images are corrected using the feature points, and then the 128D feature vectors are extracted from the images through the improved residual network. The average value is calculated for all samples, and the average value is used as the feature vector. The character label is marked and stored as data in the local face database. Specifically:

[0041] (1) For a target person, collect dozens of sample images of the person from different angles.

[0042] (2) The sample images are passed through a layered detection network to detect faces of different sizes.

[0043] Among them, the hierarchical detection network is based on the YOLO network, and different detection modules are introduced in different depth convolutional layers to detect faces of different sizes.

[0044] The structure of the hierarchical detection network is as follows Figure 2 As shown in the figure, there are three detection modules. Max-Pooling is used between the two detection modules D2 and D3 for detecting larger face sizes to increase the receptive field to detect faces of larger sizes. The detection module D1 for detecting smaller face sizes fuses the features obtained from the two convolutional layers by upsampling to detect small-sized faces.

[0045] Each detection module consists of a convolutional binary classifier and a regressor. The step sizes of detection modules D1, D2, and D3 are set to 8, 16, and 32, respectively. This can effectively improve the problem of low accuracy of Dlib for small-size face recognition.

[0046] In this embodiment, the receptive field is enlarged to ensure that information is not ignored when making decisions. l k-1 is the size of the receptive field of the k-1th layer, l k is the size of the receptive field of the kth layer, s i is the step size of the i-th convolutional layer. By increasing the number of network layers, the receptive field is increased, and two 3*3 convolutions are used to replace large-size convolutions to reduce the amount of calculation.

[0047] (3) Use the Dlib cascade regression tree training model and use a series of face images with 68 feature points calibrated to obtain labeled training data as a set of training regression trees to obtain a trained model.

[0048] (4) Use the trained model to detect 68 facial feature points of the target person in the face database, and correct the face through the feature points.

[0049] (5) Generate a 128D feature vector for the aligned face image through resnet-34. Calculate the average value of the feature vectors generated for all images of the person, and store the average value as the feature vector of the person together with the person label in the face database.

[0050] 2. Extract video frames from the video stream, detect faces in each frame, mark the face positions, and use the trained model to analyze 68 feature points. Use the feature points to correct the face. Specifically:

[0051] (1) The incoming video data is processed frame by frame. The video frames are grayed out and the processed images are detected using the above-mentioned hierarchical detection network to determine the position of the face sample in the image.

[0052] (2) Using the trained recognition model, the 68 feature points of all faces in the image are predicted for facial correction.

[0053] 3. Use the improved residual network (new-ResNet-18) to generate feature vectors for faces, where Figure 3 As shown in the figure, the improved residual network adds a feature enhancement layer at the front end of the network to enhance the expressiveness of the features and complete downsampling. The feature enhancement layer has a step size of 1 and consists of a 7*7 convolution with 16 channels and a BasicBlock with 16 channels, followed by a BasicBlock with a step size of 2 and 32 channels. This achieves the purpose of enhancing features and improving the detection accuracy of small-sized portraits.

[0054] By using the above improved residual network to extract features from face images, a 128D feature vector is obtained, which can enhance the feature expression ability.

[0055] The obtained eigenvector is converted into a column vector and recorded as b.

[0056] The feature vectors in the face database are used as row vectors and sorted by character labels to form a feature matrix A. n×128 , where n is the number of characters in the feature library.

[0057] Compute the cosine similarity of the eigenvectors in the feature matrix:

[0058]

[0059] Among them: diag() means taking the diagonal elements of the matrix to form a column vector; sim is an n×1 vector.

[0060] Based on the calculated cosine similarity value, get the most similar person:

[0061] When sim(j) is greater than the set threshold, it is considered that the person needs to be censored.

[0062] In this embodiment, it is assumed that the target person will only appear once in the same frame, so when the target person is detected, the comparison of the remaining faces in the current frame is stopped, which can improve efficiency and reduce the amount of calculation.

[0063] 4. According to the person to be coded detected in the above steps, use the face frame to locate the face position and code it. The coding effect is as follows Figure 4 shown.

[0064] The coding method can be selected as given image coding or regular mosaic.

[0065] The conventional mosaic face frame is a rectangular area, and the mosaic block size is 1 / 100 of the face frame area to ensure the coding effect.

[0066] Image Mosaic stretches the given image to the size of the obtained rectangular frame according to its length and width, and then covers it on the rectangle to cover it.

[0067] 5. Repeat steps 2 to 4 for each frame of the video to determine whether there is a target person who needs to be coded. Finally, synthesize the video stream frame by frame and output it, thus completing the automatic coding of the specific target person in the video.

Claims

1. A method for coding a specific target person in a video, characterized in that: include: (1) Build a target person face database; (2) Extract video frames from the video stream, perform face detection, position marking, and use the trained model to perform face correction; specifically, grayscale the incoming video frames, use a hierarchical detection network to perform face detection on the processed images, and determine the position of the face sample in the image; The hierarchical detection network is a YOLO network comprising three detection modules, wherein the detection module is composed of a convolutional binary classifier and a regressor, and the step sizes of the three detection modules are respectively set to 32, 16, and 8 to detect faces of different sizes; Among the three detection modules, the receptive field is increased by Max-Pooling between the two modules for detecting larger face sizes, so as to detect faces of larger sizes; The module for detecting smaller face sizes fuses the features obtained from the two convolutional layers by upsampling to detect small-sized faces; (3) extracting a feature vector from the corrected face image using an improved residual network, and comparing it with the feature vector of the target person in the face database. If the similarity is greater than a preset threshold, the face image is marked as the target person that needs to be coded; the improved residual network is to add a feature enhancement layer to the front end of the residual network to enhance the expressiveness of the feature and complete downsampling; the feature enhancement layer is composed of a 7*7 convolution with 16 channels and a BasicBlock with 16 channels, followed by a BasicBlock with a step size of 2 and 32 channels; (4) Coding the marked target person; (5) After repeating operations (2) to (4) for each frame of the video, the video stream is synthesized frame by frame and output.

2. The method for coding a specific target person in a video according to claim 1, characterized in that: The method for constructing the target person face database in step (1) includes: Collect sample images of the target person; Use the trained model to detect 68 feature points of the sample image and use the feature points to correct the image; Extract 128D feature vector from the rectified image through the improved residual network; The average value of the feature vectors generated by all sample images is calculated and stored in the database as the feature vector of the target person.

3. The method for coding a specific target person in a video according to claim 2, characterized in that: The trained model is a Dlib cascade regression tree model.

4. The method for coding a specific target person in a video according to claim 1, characterized in that: The step (3) specifically includes: Convert the 128D feature vector of the face of the person in the video frame into a column vector, denoted as b; The facial feature vectors in the face database are used as row vectors and sorted by character labels to form a feature matrix A. n×128 , where n is the number of characters in the feature library; Calculate vector similarity: Among them: diag() means taking the diagonal elements of the matrix to form a column vector; sim is an n×1 vector; Get the most similar people: When sim(j) is greater than the set threshold, it is considered to be a target person that needs to be coded.

5. The method for coding a specific target person in a video according to any one of claims 1 to 4, characterized in that: The coding process includes selecting a given picture to cover the face or blurring the face.

Images